Validate file extensions and file size on Drupal 7 file field using the form API

it helps some times You can use file_validate_size() to check the file's size before calling file_save_upload()
Here are a few examples of these functions in use:

I wish did fix the issue. Oke, I was having this problem for the last three days. Now after a few hours after I posted this question I solved the problem.
For those who have the same question or problem here is what solved it for me and what the right way is to add settings. The settings can be different for each instance so the settings go at the instance creation and not the field itself.

$instance = array(
    'field_name'    => 'file'
    'entity_type'   => 'node',
    'bundle'        => 'article',
    'label'         => 'Attachment',
    'description'   => 'Add an attachment here',
    'required'      => TRUE,
    'settings'      => array(
         'max_filesize'    => '512',
         'file_extensions' => 'zip txt pdf docx'
    ),
);
field_create_instance($instance);

seems to work fine The answer is to change value of file_extensions to jpg png gif pdf zip doc rtf xdoc rar txt.

this one helps. I got the answer after digging drupal 8 core APIs. If we're using a file field, then we can set the "file_extensions" setting. You can find out more information about a "file" type by going to the FileItem API.

$fields['my_new_file'] = BaseFieldDefinition::create('file')
   ->setSetting('file_extensions', 'xls xlsx');

this will help Here's a generic Vue example of how to validate the file's size before the form is submitted.
The crux is obtaining the file object from the files property on the input itself, and checking the file's size via the size property; the rest is just stuff related to preventing the form from being submitted if the validation fails.

new Vue({
  el: '#app',
  methods: {
    onSubmit(e) {
      const file = this.$refs.file.files[0];
      
      if (!file) {
        e.preventDefault();
        alert('No file chosen');
        return;
      }
      
      if (file.size > 1024 * 1024) {
        e.preventDefault();
        alert('File too big (> 1MB)');
        return;
      }
      
      alert('File OK');
    },
  },
});

<script src="https://rawgit.com/vuejs/vue/dev/dist/vue.js"></script>

<div id="app">
  <form @submit="onSubmit">
    <input type="file" ref="file">
    <button type="submit">Submit</button>
  </form>
</div>