Perhaps transmission-create may encounter some issues when creating a torrent for large files.

[cosikng]

What is the issue?

What's the problem?

I've been experiencing some issues lately while creating torrents using Transmission. After creating the torrent with transmission-create and starting seeding, Transmission begins to verify the local files. However, it always gets stuck at around 99%. No matter how many times I retry 'Verify Local Data', it remains stuck. As a result, I had to recreate the torrent several times, leading me to delete the torrent already uploaded to the PT website. However, the problem still persists. This has caused me a lot of inconvenience. There are 78 files I want to seed, totaling approximately 30GB.

What did I do to investigate the issue?

To investigate what was happening, I used transmission-create to create two torrents, both with a block size of 2048KB. I saved both of these torrents, distinguishing one of them by adding the '_new' suffix.
transmission-create -s 2048 -o mytorrent.torrent /path/to/myfile
Afterward, I wrote a simple Python script to extract the 'pieces' field from both of these torrents, which contains the hash values for all files.

import bencoder


def extract_pieces(torrent_file_path, output_file_path):
    try:
        with open(torrent_file_path, 'rb') as f:
            data = f.read()

        torrent_data = bencoder.bdecode(data)


        pieces_data = torrent_data[b'info'][b'pieces']


        with open(output_file_path, 'wb') as f:
            f.write(pieces_data)

        # print(pieces_data)

    except Exception as e:
        print(e)



torrent_file_path = 'mytorrent_new.torrent' 
output_file_path = 'mytorrent_new.bin'  
extract_pieces(torrent_file_path, output_file_path)

Then, I performed SHA256 checksums on these two extracted binary files and found them to be different. Furthermore, I used the following script to compare the extracted hash values byte by byte to identify the differing positions.

def compare_bytes(byte_str1, byte_str2):
    if len(byte_str1) != len(byte_str2):
        raise ValueError("Length not match")

    diff_indices = []
    for i in range(len(byte_str1)):
        if byte_str1[i] != byte_str2[i]:
            diff_indices.append(i)

    return diff_indices


def display_difference(byte_str1, byte_str2, diff_indices):
    if not diff_indices:
        print("No difference")
    else:
        print("Difference:")
        for idx in diff_indices:
            print(f"Pos {idx}: {byte_str1[idx]} -> {byte_str2[idx]}")


with open('mytorrent.bin', 'rb') as f:
    byte_str1 = f.read(-1)
with open('mytorrent_new.bin', 'rb') as f:
    byte_str2 = f.read(-1)

diff_indices = compare_bytes(byte_str1, byte_str2)
display_difference(byte_str1, byte_str2, diff_indices)

I found that approximately three hundred blocks' hash values were different, which is roughly the size of one file. This torrent has a total of about 17,000 blocks.

I speculate that this might be an error occurring at certain positions during the creation of the torrent file by transmission-create. I am currently unable to determine whether this is an internal error of the program or a disk read/write error. However, I believe that even if it's a disk read/write error, transmission-create should be able to handle it correctly.

Out of consideration for community rules, I feel it's inappropriate to share my torrent files. However, I'm more than willing to provide any additional information if needed. I appreciate any help and advice.

Which application of Transmission?

transmission-daemon

Which version of Transmission?

4.0.5

[cosikng]

Correction:

I apologize for the error in counting the differing hash values between the two torrent files. The actual number of discrepancies should be around 15 blocks. Based on further calculations, this aligns well with the number of erroneous blocks encountered during local data verification within Transmission.

[tearfur]

I appreciate your efforts to investigate this.

It's really hard for us to debug without any specifics though. I have some questions:

1. Does this only happen with transmission-create? Or other torrent creators as well?
2. Is there any public domain file where this can be reproduced?

[cosikng]

I am using an embedded Linux system (similar to a Raspberry Pi) as a seed server, running Transmission on it. I haven't tried other programs for creating torrents on it. I am still trying to reproduce the hash value error that occurs when creating a torrent. For example, I tried using the dd command to fully load my disk while creating a torrent with transmission-create, but I haven't found a consistent method to reproduce the issue. If it would be helpful, I can share the torrents that were created incorrectly and the ones that were created correctly (without tracker information and not distributed on DHT). If there is anything else you need me to provide, please let me know.

[tearfur]

The resulting torrent files isn't very useful for us to reproduce the bug. Ideally we'd want a file/folder that can trigger the bug.

Also could you please check if other torrent creators has problems as well?

We've had a few other reports of similar nature, but none of them could provide any concrete information for debugging unfortunately:

• transmission-create reads input files with incorrect size #6289
• CLI : transmission-create command create a corrupted .torrent file #6609

[cosikng]

I tried copying all the files to my Windows computer and used qBittorrent to create the torrent, and everything worked fine. What I want to say is that this issue may not be strongly related to specific files or folders. Over the past two days, I tried creating torrents with transmission-create for the same folder about dozens of times, but I only managed to reproduce the issue (i.e., hash value errors in some blocks) once. All other attempts were normal.

The clue I can provide is that when I initially created the problematic torrent file, my hard drive load seemed to be very high: several seeding files were being uploaded, and Jellyfin (a home media streaming service) was scanning some newly added libraries. This is why I tried using the dd command to fully load the hard drive to reproduce the issue.

I have a hypothesis: could this issue be caused by transmission-create encountering very high disk read/write latency during torrent creation? After all, my device is connected to a relatively poor-performing mechanical hard drive.

[Pentaphon]

I have a hypothesis: could this issue be caused by transmission-create encountering very high disk read/write latency during torrent creation? After all, my device is connected to a relatively poor-performing mechanical hard drive.

No way to tell unless you provide a file for us or upgrade your hardware to SSD.

[tearfur]

I have a hypothesis: could this issue be caused by transmission-create encountering very high disk read/write latency during torrent creation? After all, my device is connected to a relatively poor-performing mechanical hard drive.

I don't see how that can happen... AFAICT we only do synchronised read/write.