Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage Emulator Upload Error: Range invalid for upload data. #4364

Closed
Vortec4800 opened this issue Mar 26, 2022 · 4 comments
Closed

Storage Emulator Upload Error: Range invalid for upload data. #4364

Vortec4800 opened this issue Mar 26, 2022 · 4 comments

Comments

@Vortec4800
Copy link

Vortec4800 commented Mar 26, 2022

[REQUIRED] Environment info

firebase-tools: 10.5.0

Platform: macOS 12.3 on M1 Max

[REQUIRED] Test case

private func uploadSelectedImage(_ selectedImage: UIImage) async {
	do {
		let storageRef = Storage.storage().reference().child("gallery/image.jpg")
		let metadata = StorageMetadata()
		metadata.contentType = "image/jpg"
		if let data = selectedImage.jpegData(compressionQuality: 0.8) {
			let uploadMetadata = try await storageRef.putDataAsync(data, metadata: metadata)
			
			print("\(uploadMetadata)")
		}
	} catch {
		print("Error uploading image: \(error)")
	}
}

[REQUIRED] Steps to reproduce

Upload an image using the provided function, after connecting to Firebase Storage Emulator.

Storage rules are to allow writes if authenticated.

rules_version = '2';
service firebase.storage {
  match /b/{bucket}/o {
    match /{allPaths=**} {
      allow read, write: if request.auth!=null;
    }
  }
}

[REQUIRED] Expected behavior

Image to upload to Storage Emulator and print uploadMetadata to console.

[REQUIRED] Actual behavior

Image actually does successfully upload looking in the Storage Emulator UI, however the error catch happens with the following output:

2022-03-26 15:10:39.827585-0500 ConnectDataTest[13671:170319] chunk fetcher completion has kStatusUnknown upload status for headers <CFBasicHash 0x6000035fb600 [0x1de715af0]>{type = immutable dict, count = 9,
entries =>
	0 : Connection = <CFString 0x6000020ed560 [0x1de715af0]>{contents = "keep-alive"}
	1 : X-Powered-By = Express
	2 : Access-Control-Expose-Headers = <CFString 0x10a5423d0 [0x1de715af0]>{contents = "content-type,x-firebase-storage-version,x-goog-upload-url,x-goog-upload-status,x-goog-upload-command,x-gupload-uploadid,x-goog-upload-header-content-length,x-goog-upload-header-content-type,x-goog-upload-protocol,x-goog-upload-status,x-goog-upload-chunk-granularity,x-goog-upload-control-url"}
	3 : Content-Type = <CFString 0x600003594940 [0x1de715af0]>{contents = "application/json; charset=utf-8"}
	4 : Etag = <CFString 0x600003594b80 [0x1de715af0]>{contents = "W/"203-ubgKnIPuC7Bbtb6Rw8XCe2hadgc""}
	6 : Date = <CFString 0x600002e2eb80 [0x1de715af0]>{contents = "Sat, 26 Mar 2022 20:10:39 GMT"}
	10 : Content-Length = 515
	11 : Keep-Alive = <CFString 0x6000020ed620 [0x1de715af0]>{contents = "timeout=5"}
	12 : Vary = Origin
}
 fetcher GTMSessionUploadFetcher 0x12a518200 (http://localhost:9199/v0/b/connect-2022-2815c.appspot.com/o/gallery%2Fimage.jpg?uploadType=resumable&name=gallery%2Fimage.jpg)
2022-03-26 15:10:39.828028-0500 ConnectDataTest[13671:170319] uploadStatus:(null)  newOffset:2590798 (0 + 2590798)  fullUploadLength:2590798 chunkFetcher:GTMSessionFetcher 0x105008c50 (http://localhost:9199/v0/b/connect-2022-2815c.appspot.com/o?name=gallery/image.jpg&upload_id=6f87b149-88d9-4e23-8221-126763681690&upload_protocol=resumable) requestHeaders:<CFBasicHash 0x600003515dc0 [0x1de715af0]>{type = mutable dict, count = 5,
entries =>
	0 : User-Agent = <CFString 0x600002e21920 [0x1de715af0]>{contents = "com.cityelectricsupply.connect2022/1.0 iPhone/15.4 hw/sim (GTMSUF/1)"}
	2 : X-Goog-Upload-Offset = 0
	3 : Content-Length = 2590798
	5 : X-Goog-Upload-Protocol = <CFString 0x1012eb6f8 [0x1de715af0]>{contents = "resumable"}
	6 : X-Goog-Upload-Command = <CFString 0x1012ebd58 [0x1de715af0]>{contents = "upload, finalize"}
}
 responseHeaders:<CFBasicHash 0x6000035fb600 [0x1de715af0]>{type = immutable dict, count = 9,
entries =>
	0 : Connection = <CFString 0x6000020ed560 [0x1de715af0]>{contents = "keep-alive"}
	1 : X-Powered-By = Express
	2 : Access-Control-Expose-Headers = <CFString 0x10a5423d0 [0x1de715af0]>{contents = "content-type,x-firebase-storage-version,x-goog-upload-url,x-goog-upload-status,x-goog-upload-command,x-gupload-uploadid,x-goog-upload-header-content-length,x-goog-upload-header-content-type,x-goog-upload-protocol,x-goog-upload-status,x-goog-upload-chunk-granularity,x-goog-upload-control-url"}
	3 : Content-Type = <CFString 0x600003594940 [0x1de715af0]>{contents = "application/json; charset=utf-8"}
	4 : Etag = <CFString 0x600003594b80 [0x1de715af0]>{contents = "W/"203-ubgKnIPuC7Bbtb6Rw8XCe2hadgc""}
	6 : Date = <CFString 0x600002e2eb80 [0x1de715af0]>{contents = "Sat, 26 Mar 2022 20:10:39 GMT"}
	10 : Content-Length = 515
	11 : Keep-Alive = <CFString 0x6000020ed620 [0x1de715af0]>{contents = "timeout=5"}
	12 : Vary = Origin
}
2022-03-26 15:10:39.828459-0500 ConnectDataTest[13671:170319] offset 2590798 exceeds data length 2590798
2022-03-26 15:10:39.828531-0500 ConnectDataTest[13671:170319] Range invalid for upload data.  offset: 2590798	length: 798	dataLength: 2590798
2022-03-26 15:18:38.822544-0500 ConnectDataTest[13671:170451] [connection] nw_socket_output_finished [C2.2:2] shutdown(21, SHUT_WR) [57: Socket is not connected]
3/26/22, 3:18:38 PM CDT PhotoUploadTestViewModel line 44 $ Error uploading image: Error Domain=FIRStorageErrorDomain Code=-13000 "An unknown error occurred, please check the server response." UserInfo={NSLocalizedDescription=An unknown error occurred, please check the server response., ResponseErrorDomain=com.google.GTMSessionFetcher, object=gallery/image.jpg, description=Range invalid for upload data.  offset: 2590798	length: 798	dataLength: 2590798, ResponseErrorCode=-2, bucket=connect-2022-2815c.appspot.com}

The error output:

Error Domain=FIRStorageErrorDomain Code=-13000 "An unknown error occurred, please check the server response." UserInfo={NSLocalizedDescription=An unknown error occurred, please check the server response., ResponseErrorDomain=com.google.GTMSessionFetcher, object=gallery/image.jpg, description=Range invalid for upload data.  offset: 2590798	length: 798	dataLength: 2590798, ResponseErrorCode=-2, bucket=connect-2022-2815c.appspot.com}

States to check server output, however the server console logs don't say anything interesting. No console logs happen at all when the storage upload trigger is called.

$ firebase emulators:start --inspect-functions
⚠  emulators: Support for Java version <= 10 will be dropped soon in firebase-tools@11. Please upgrade to Java version 11 or above to continue using the emulators.
i  emulators: Starting emulators: auth, functions, firestore, storage
⚠  functions: You are running the Functions emulator in debug mode (port=9229). This means that functions will execute in sequence rather than in parallel.
⚠  functions: The following emulators are not running, calls to these services from the Functions emulator will affect production: database, hosting, pubsub
✔  functions: Using node@16 from host.
i  firestore: Firestore Emulator logging to firestore-debug.log
i  ui: Emulator UI logging to ui-debug.log
i  functions: Watching "/Users/cimdieke/Projects/Connect2022-Firebase/functions" for Cloud Functions...
✔  functions[us-central1-auth-createUserProfile]: auth function initialized.
✔  functions[us-central1-func-makeUppercase]: firestore function initialized.

┌─────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! It is now safe to connect your app. │
│ i  View Emulator UI at http://localhost:4000                │
└─────────────────────────────────────────────────────────────┘

┌────────────────┬────────────────┬─────────────────────────────────┐
│ Emulator       │ Host:Port      │ View in Emulator UI             │
├────────────────┼────────────────┼─────────────────────────────────┤
│ Authentication │ localhost:9099 │ http://localhost:4000/auth      │
├────────────────┼────────────────┼─────────────────────────────────┤
│ Functions      │ localhost:5001 │ http://localhost:4000/functions │
├────────────────┼────────────────┼─────────────────────────────────┤
│ Firestore      │ localhost:8086 │ http://localhost:4000/firestore │
├────────────────┼────────────────┼─────────────────────────────────┤
│ Storage        │ localhost:9199 │ http://localhost:4000/storage   │
└────────────────┴────────────────┴─────────────────────────────────┘
  Emulator Hub running at localhost:4400
  Other reserved ports: 4500

Issues? Report them at https://github.com/firebase/firebase-tools/issues and attach the *-debug.log files.
 
⚠  emulators: Support for Java version <= 10 will be dropped soon in firebase-tools@11. Please upgrade to Java version 11 or above to continue using the emulators.
>  Debugger listening on ws://localhost:9229/b1981fd9-abca-4f8f-a51d-dd9f5318e1da
>  Debugger listening on ws://localhost:9229/b1981fd9-abca-4f8f-a51d-dd9f5318e1da
>  For help, see: https://nodejs.org/en/docs/inspector
@russellwheatley
Copy link
Member

Hey, I'd just to chime in by saying the FlutterFire CI for iOS is now breaking with the above mentioned error. We're also using the latest firebase-tools (version 10.5.0).

To see this error in action, do the following:

  1. Create throwaway flutter app (flutter create throwawayProject).
  2. Copy paste the below noted code in "lib/main.dart" file and insert your Firebase project details as specified by the code comment.
  3. Run Firebase storage emulator with the below noted Firebase Storage rules.
  4. Run Flutter project on an iOS simulator and observe the error appear in your console as you try to upload a string or file. If you run the Flutter project on an android emulator, it will work as expected.

Code to copy/pasted into "lib/main.dart" file:

import 'dart:async';
import 'dart:io' as io;

import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_storage/firebase_storage.dart';
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:image_picker/image_picker.dart';

import 'firebase_options.dart';
import 'save_as/save_as.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp(
  	// <INSERT YOUR FIREBASE APP INFORMATION HERE>
    options: FirebaseOptions(apiKey: '', appId: '', messagingSenderId: '', projectId: ''),
  );

  final emulatorHost =
      (!kIsWeb && defaultTargetPlatform == TargetPlatform.android)
          ? '10.0.2.2'
          : 'localhost';

  await FirebaseStorage.instance.useStorageEmulator(emulatorHost, 9199);

  runApp(StorageExampleApp());
}

/// Enum representing the upload task types the example app supports.
enum UploadType {
  /// Uploads a randomly generated string (as a file) to Storage.
  string,

  /// Uploads a file from the device.
  file,

  /// Clears any tasks from the list.
  clear,
}

/// The entry point of the application.
///
/// Returns a [MaterialApp].
class StorageExampleApp extends StatelessWidget {
  StorageExampleApp({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
        title: 'Storage Example App',
        theme: ThemeData.dark(),
        home: Scaffold(
          body: TaskManager(),
        ));
  }
}

/// A StatefulWidget which keeps track of the current uploaded files.
class TaskManager extends StatefulWidget {
  // ignore: public_member_api_docs
  TaskManager({Key? key}) : super(key: key);

  @override
  State<StatefulWidget> createState() {
    return _TaskManager();
  }
}

class _TaskManager extends State<TaskManager> {
  List<UploadTask> _uploadTasks = [];

  /// The user selects a file, and the task is added to the list.
  Future<UploadTask?> uploadFile(XFile? file) async {
    if (file == null) {
      ScaffoldMessenger.of(context).showSnackBar(const SnackBar(
        content: Text('No file was selected'),
      ));

      return null;
    }

    UploadTask uploadTask;

    // Create a Reference to the file
    Reference ref = FirebaseStorage.instance
        .ref()
        .child('playground')
        .child('/some-image.jpg');

    final metadata = SettableMetadata(
        contentType: 'image/jpeg',
        customMetadata: {'picked-file-path': file.path});

    if (kIsWeb) {
      uploadTask = ref.putData(await file.readAsBytes(), metadata);
    } else {
      uploadTask = ref.putFile(io.File(file.path), metadata);
    }

    return Future.value(uploadTask);
  }

  /// A new string is uploaded to storage.
  UploadTask uploadString() {
    const String putStringText =
        'This upload has been generated using the putString method! Check the metadata too!';

    // Create a Reference to the file
    Reference ref = FirebaseStorage.instance
        .ref()
        .child('playground')
        .child('/put-string-example.txt');

    // Start upload of putString
    return ref.putString(putStringText,
        metadata: SettableMetadata(
            contentLanguage: 'en',
            customMetadata: <String, String>{'example': 'putString'}));
  }

  /// Handles the user pressing the PopupMenuItem item.
  Future<void> handleUploadType(UploadType type) async {
    switch (type) {
      case UploadType.string:
        setState(() {
          _uploadTasks = [..._uploadTasks, uploadString()];
        });
        break;
      case UploadType.file:
        final file = await ImagePicker().pickImage(source: ImageSource.gallery);
        UploadTask? task = await uploadFile(file);

        if (task != null) {
          setState(() {
            _uploadTasks = [..._uploadTasks, task];
          });
        }
        break;
      case UploadType.clear:
        setState(() {
          _uploadTasks = [];
        });
        break;
    }
  }

  void _removeTaskAtIndex(int index) {
    setState(() {
      _uploadTasks = _uploadTasks..removeAt(index);
    });
  }

  Future<void> _downloadBytes(Reference ref) async {
    final bytes = await ref.getData();
    // Download...
    await saveAsBytes(bytes!, 'some-image.jpg');
  }

  Future<void> _downloadLink(Reference ref) async {
    final link = await ref.getDownloadURL();

    await Clipboard.setData(ClipboardData(
      text: link,
    ));

    ScaffoldMessenger.of(context).showSnackBar(
      const SnackBar(
        content: Text(
          'Success!\n Copied download URL to Clipboard!',
        ),
      ),
    );
  }

  Future<void> _downloadFile(Reference ref) async {
    final io.Directory systemTempDir = io.Directory.systemTemp;
    final io.File tempFile = io.File('${systemTempDir.path}/temp-${ref.name}');
    if (tempFile.existsSync()) await tempFile.delete();

    await ref.writeToFile(tempFile);

    ScaffoldMessenger.of(context).showSnackBar(
      SnackBar(
        content: Text(
          'Success!\n Downloaded ${ref.name} \n from bucket: ${ref.bucket}\n '
          'at path: ${ref.fullPath} \n'
          'Wrote "${ref.fullPath}" to tmp-${ref.name}',
        ),
      ),
    );
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('Storage Example App'),
        actions: [
          PopupMenuButton<UploadType>(
            onSelected: handleUploadType,
            icon: const Icon(Icons.add),
            itemBuilder: (context) => [
              const PopupMenuItem(
                  // ignore: sort_child_properties_last
                  child: Text('Upload string'),
                  value: UploadType.string),
              const PopupMenuItem(
                  // ignore: sort_child_properties_last
                  child: Text('Upload local file'),
                  value: UploadType.file),
              if (_uploadTasks.isNotEmpty)
                const PopupMenuItem(
                    // ignore: sort_child_properties_last
                    child: Text('Clear list'),
                    value: UploadType.clear)
            ],
          )
        ],
      ),
      body: _uploadTasks.isEmpty
          ? const Center(child: Text("Press the '+' button to add a new file."))
          : ListView.builder(
              itemCount: _uploadTasks.length,
              itemBuilder: (context, index) => UploadTaskListTile(
                task: _uploadTasks[index],
                onDismissed: () => _removeTaskAtIndex(index),
                onDownloadLink: () async {
                  return _downloadLink(_uploadTasks[index].snapshot.ref);
                },
                onDownload: () async {
                  if (kIsWeb) {
                    return _downloadBytes(_uploadTasks[index].snapshot.ref);
                  } else {
                    return _downloadFile(_uploadTasks[index].snapshot.ref);
                  }
                },
              ),
            ),
    );
  }
}

/// Displays the current state of a single UploadTask.
class UploadTaskListTile extends StatelessWidget {
  // ignore: public_member_api_docs
  const UploadTaskListTile({
    Key? key,
    required this.task,
    required this.onDismissed,
    required this.onDownload,
    required this.onDownloadLink,
  }) : super(key: key);

  /// The [UploadTask].
  final UploadTask /*!*/ task;

  /// Triggered when the user dismisses the task from the list.
  final VoidCallback /*!*/ onDismissed;

  /// Triggered when the user presses the download button on a completed upload task.
  final VoidCallback /*!*/ onDownload;

  /// Triggered when the user presses the "link" button on a completed upload task.
  final VoidCallback /*!*/ onDownloadLink;

  /// Displays the current transferred bytes of the task.
  String _bytesTransferred(TaskSnapshot snapshot) {
    return '${snapshot.bytesTransferred}/${snapshot.totalBytes}';
  }

  @override
  Widget build(BuildContext context) {
    return StreamBuilder<TaskSnapshot>(
      stream: task.snapshotEvents,
      builder: (
        BuildContext context,
        AsyncSnapshot<TaskSnapshot> asyncSnapshot,
      ) {
        Widget subtitle = const Text('---');
        TaskSnapshot? snapshot = asyncSnapshot.data;
        TaskState? state = snapshot?.state;

        if (asyncSnapshot.hasError) {
          if (asyncSnapshot.error is FirebaseException &&
              // ignore: cast_nullable_to_non_nullable
              (asyncSnapshot.error as FirebaseException).code == 'canceled') {
            subtitle = const Text('Upload canceled.');
          } else {
            // ignore: avoid_print
            print(asyncSnapshot.error);
            subtitle = const Text('Something went wrong.');
          }
        } else if (snapshot != null) {
          subtitle = Text('$state: ${_bytesTransferred(snapshot)} bytes sent');
        }

        return Dismissible(
          key: Key(task.hashCode.toString()),
          onDismissed: ($) => onDismissed(),
          child: ListTile(
            title: Text('Upload Task #${task.hashCode}'),
            subtitle: subtitle,
            trailing: Row(
              mainAxisSize: MainAxisSize.min,
              children: <Widget>[
                if (state == TaskState.running)
                  IconButton(
                    icon: const Icon(Icons.pause),
                    onPressed: task.pause,
                  ),
                if (state == TaskState.running)
                  IconButton(
                    icon: const Icon(Icons.cancel),
                    onPressed: task.cancel,
                  ),
                if (state == TaskState.paused)
                  IconButton(
                    icon: const Icon(Icons.file_upload),
                    onPressed: task.resume,
                  ),
                if (state == TaskState.success)
                  IconButton(
                    icon: const Icon(Icons.file_download),
                    onPressed: onDownload,
                  ),
                if (state == TaskState.success)
                  IconButton(
                    icon: const Icon(Icons.link),
                    onPressed: onDownloadLink,
                  ),
              ],
            ),
          ),
        );
      },
    );
  }
}

Storage emulator rules:

rules_version = '2';
service firebase.storage {
  match /b/{bucket}/o {
    match /{document=**} {
      allow read, write: if false;
    }
  match /playground/{document=**} {
      allow read, write: if true;
    }
  }
}

@Vortec4800
Copy link
Author

I edited my original post to add this info but wanted to comment as well in case it gets missed and may be important, I'm running the emulator suite on my M1 Max laptop.

@built-by-as
Copy link

I'm also running into the same error

2022-03-31 16:50:05.822789-0400 Example[51380:621771] offset 438491 exceeds data length 438491
2022-03-31 16:50:05.823434-0400 Example[51380:622628] Range invalid for upload data.  offset: 438491	length: 8491	dataLength: 438491	expected UploadLength: 438491
2022-03-31 16:50:08.017868-0400 Example[51380:622083] [javascript] Possible Unhandled Promise Rejection (id: 0):

@tohhsinpei
Copy link
Member

Fixed in #4407

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants