Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix newlib #3345

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Fix newlib #3345

wants to merge 5 commits into from

Conversation

zetanumbers
Copy link

@zetanumbers zetanumbers commented Sep 8, 2023

So i found several incorrect definitions in the newlib module. I cannot trace these values to anything except #646 which was done for devkitpro, yet definitions there don't seem to be different from fixed ones. devkitPro/libctru has these wrong values.

PR checks
  • Edit corresponding file(s) under libc-test/semver when you add/remove item(s)
  • Your PR doesn't contain any unstable values like *LAST or *MAX (see #3131)
  • If your PR increments version number, it must not contain any other changes
  • rustc ci/style.rs && ./style src
  • cd libc-test && cargo test
    • (this might fail on your env due to environment difference between your env and CI. Ignore failures if you are not sure)

I'm not sure how to do all of these.

@rustbot
Copy link
Collaborator

rustbot commented Sep 8, 2023

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @JohnTitor (or someone else) soon.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

  • @rustbot author: the review is finished, PR author should check the comments and take action accordingly
  • @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

src/unix/newlib/mod.rs Show resolved Hide resolved
pub type mode_t = ::c_uint;
pub type nfds_t = u32;
} else {
pub type mode_t = u32;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto for mode_t and nfds_t?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mode_t:

https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/include/sys/_types.h;h=d1112d5d102ca509331b9db5b27934b7cc403b32;hb=9e09d6ed83cce4777a5950412647ccc603040409#l90

DOS newlib isn't used or supported for rust by anyone, so i didn't that definition.
Couldn't figure out how to specify cfg attributes for SPARC definition due to __srv4__. However i haven't seen (even custom) rust target that would use that.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nfds_t:

https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/sys/rtems/include/sys/poll.h;h=cc6ad49db033dfc63714d5062da30bf08b364dc3;hb=9e09d6ed83cce4777a5950412647ccc603040409#l42

Not sure about newlib structure of rtems, but i found it only there. Found it only there, doesn't look right, may revert.

pub type nlink_t = ::c_ushort;
pub type pthread_t = ::c_ulong;

cfg_if! {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto for pthread_t?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -557,16 +585,21 @@ pub const TCP_KEEPCNT: ::c_int = 1024;
cfg_if! {
if #[cfg(target_os = "horizon")] {
pub const IP_TOS: ::c_int = 7;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

src/unix/newlib/mod.rs Show resolved Hide resolved
src/unix/newlib/mod.rs Show resolved Hide resolved
Copy link
Contributor

@Meziu Meziu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As one of the main maintainers of the armv6k-nintendo-3ds target and owner of @rust3ds, I suggested some changes required to keep compatibility with devkitPro's custom implementation of newlib and libctru.

When I wrote the base horizon.rs definitions for libc I noticed how weirdly those were handled in the upstream libraries, which (due to my rather unwise implementation) propagated some errors to the other members of the newlib module, such as esp-idf (sorry!).

Now that the module seems to be getting more attention, we should strive to work together to ensure correct behavior on all platforms. I'll personally see what I can do in regards of the custom definitions for horizonOS.

Still, thank you for correctly handling most of the changes already, only a couple of errors are left for what concerns horizonOS. 👍

pub const EAI_FAMILY: ::c_int = -303;
pub const EAI_MEMORY: ::c_int = -304;
pub const EAI_NONAME: ::c_int = -305;
pub const EAI_SOCKTYPE: ::c_int = -307;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These definitions must be included via a cfg_if! for horizonOS. All other EAI definitions can be kept unified for all platforms.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. This was from past commit, so i haven't yet tried to preserve these mixed newlib and libctru values. You confirmed for me that this is the right approach. Now i can confidently finish this PR. Thank you.

@@ -546,8 +574,8 @@ pub const IFF_LINK2: ::c_int = 0x4000; // per link layer defined bit
pub const IFF_ALTPHYS: ::c_int = IFF_LINK2; // use alternate physical connection
pub const IFF_MULTICAST: ::c_int = 0x8000; // supports multicast

pub const TCP_NODELAY: ::c_int = 8193;
pub const TCP_MAXSEG: ::c_int = 8194;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These definitions must be included via a cfg_if! for horizonOS.

@ian-h-chamberlain
Copy link
Contributor

ian-h-chamberlain commented Sep 23, 2023

One other thing I think is worth mentioning: target_os = "horizon" also applies to the aarch64-nintendo-switch-freestanding target, but I'm not sure if these values need to be different on that platform as well (it doesn't look like they set env: "newlib" so maybe it is irrelevant). I don't think it would necessarily be a regression but tagging the maintainers of that platform @leo60228 @jam1garner just in case they have input for how this change might affect it.

@jam1garner
Copy link
Contributor

Freestanding using newlib is a combination that doesn't currently have any meaning, there's no port of newlib targeting freestanding environments (and I'd go as far as to say there never will be) so this shouldn't affect the freestanding target.

@bors
Copy link
Contributor

bors commented Sep 24, 2023

☔ The latest upstream changes (presumably #3284) made this pull request unmergeable. Please resolve the merge conflicts.

@pheki
Copy link
Contributor

pheki commented Sep 29, 2023

@jam1garner What about this target that's on CI for this repo? https://github.com/rust-lang/libc/blob/main/ci/switch.json

Looks like it's for the switch with newlib, added by @leo60228 4 years ago

@jam1garner
Copy link
Contributor

yeah idk leo can probably shed more light on that but my best guess is that target json is aimed at homebrew environments not freestanding

@leo60228
Copy link
Contributor

I don't really remember after four years, sorry. I don't think the configuration that .json is for is still supported, though.

@JohnTitor
Copy link
Member

Could anyone give me a summary of the discussion here? I'd like to help merge this but don't have enough context.

@zetanumbers
Copy link
Author

zetanumbers commented Jan 4, 2024

Could anyone give me a summary of the discussion here? I'd like to help merge this but don't have enough context.

A lot of wrong definitions, particularly for newlib targets. The main reason, I would say, is that it has definitions from libctru for nintendo 3DS, which are super different from the most og newlib. This is because newlib code was introduced precisely for nintendo 3DS.

I no longer have time to untangle this problem to check field offests, struct alignments and sizes, etc. I would be happy to pass this torch to anyone interested. I thought of making some instrument to generate test C code, but it bacame too unwieldy to me. I also had hard time configuring ESP32 toolchain, it being a bash shell you have to dive into before getting access to compiler, so I haven't managed to check it either.

@zetanumbers
Copy link
Author

zetanumbers commented Jan 4, 2024

Keep in mind horizon os of 3DS as to preserve these libctru definitions.

@Meziu
Copy link
Contributor

Meziu commented Jan 4, 2024

Indeed. Some wrong definitions had been introduced for newlib targets due to them being needed by horizon OS for the N3DS target (and not being present for any other newlib target). For what concerns HorizonOS, all the "current" definitions should be kept via cfg_if or by moving them into the horizon module, since they are valid in our context.

Any changes made should not disrupt our current implementation.

@zetanumbers
Copy link
Author

zetanumbers commented Jan 4, 2024

I wrote the script to check some definitions, here are results for ps vita
sizeof (struct pollfd_replica) == 12 != 8 == sizeof (struct pollfd)
sizeof (struct statvfs_replica) == 56 != 1 == sizeof (statvfs)
alignof (struct statvfs_replica) == 8 != 2 == alignof (statvfs)
sizeof (struct sigaction_replica) == 12 != 1 == sizeof (sigaction)
alignof (struct sigaction_replica) == 4 != 2 == alignof (sigaction)
sizeof (struct dirent_replica) == 344 != 352 == sizeof (struct dirent)
alignof (struct dirent_replica) == 1 != 8 == alignof (struct dirent)
sizeof (struct passwd_replica) == 24 != 28 == sizeof (struct passwd)
sizeof (struct sem_t_replica) == 16 != 4 == sizeof (sem_t)
alignof (struct sem_t_replica) == 1 != 4 == alignof (sem_t)
sizeof (struct utsname_replica) == 390 != 325 == sizeof (struct utsname)
sizeof (struct cpu_set_t_replica) == 128 != 4 == sizeof (cpu_set_t)
sizeof (struct pthread_cond_t_replica) == 8 != 4 == sizeof (pthread_cond_t)
alignof (struct pthread_cond_t_replica) == 8 != 4 == alignof (pthread_cond_t)
alignof (struct pthread_attr_t_replica) == 1 != 4 == alignof (pthread_attr_t)
alignof (struct pthread_rwlockattr_t_replica) == 1 != 4 == alignof (pthread_rwlockattr_t)
sizeof (vita-samples/hello_world/src/newlib_assert.h:409): 4 != 8
O_CLOEXEC == 262144 != 524288
PF_INET6 == 24 != 23
AI_NUMERICSERV == 8 != 0
AI_ADDRCONFIG == 1024 != 0
NI_NUMERICSERV == 8 != 0
NI_DGRAM == 16 != 0
EAI_AGAIN == -3 != 2
EAI_BADFLAGS == -1 != 3
EAI_FAIL == -4 != 4
EAI_FAMILY == -6 != 5
EAI_MEMORY == -10 != 6
EAI_NONAME == -2 != 8
EAI_SERVICE == -8 != 9
EAI_SOCKTYPE == -7 != 10
EAI_SYSTEM == -11 != 11
EAI_OVERFLOW == -12 != 14
Here's the script
#include <stdlib.h>
#include <pthread.h>
#include <errno.h>
#include <sys/socket.h>
#include <fcntl.h>
#include <time.h>
#include <netdb.h>
#include <netinet/tcp.h>
#include <netinet/in.h>
#include <sys/resource.h>
#if !defined (__3DS__)
#include <dlfcn.h>
#endif
#include <sys/unistd.h>
#include <time.h>
#include <sys/types.h>
#include <sys/utsname.h>
#include <semaphore.h>
#include <pwd.h>
#include <dirent.h>
#include <sys/statvfs.h>
#include <locale.h>
#include <sys/poll.h>
#include <sys/socket.h>
#include <netdb.h>

void assert_eq_impl(const char* name, long long got, long long expected);
#define ASSERT_EQ(C, V) assert_eq_impl(#C, C, V)

void assert_bytes_eq_impl(const char* name, long long line, const void* lhs_ptr, long long lhs_size, const void* rhs_ptr, long long rhs_size);
#define ASSERT_BYTES_EQ(A, B) assert_bytes_eq_impl(__FILE__, __LINE__, A, sizeof(*A), B, sizeof(*B))

void assert_t_eq_impl(const char* lhs_name, long long lhs_size, long long lhs_align, const char* rhs_name, long long rhs_size, long long rhs_align);
#define ASSERT_T_EQ(T, U) assert_t_eq_impl(#T, sizeof (T), _Alignof (T), #U, sizeof (U), _Alignof (U))

static void assert_definitions() {

#if defined (__vita__) | defined (__3DS__) | defined (ESP_PLATFORM)
#define POINTER_SIZE 4
#endif

  ASSERT_EQ(POINTER_SIZE, sizeof (void*));

	ASSERT_T_EQ(blkcnt_t, int32_t);
	ASSERT_T_EQ(blksize_t, int32_t);

  ASSERT_T_EQ(clockid_t, unsigned long);

#if defined (__3DS__)
  ASSERT_T_EQ(dev_t, uint32_t);
  ASSERT_T_EQ(off_t, int64_t);
#else
  ASSERT_T_EQ(dev_t, short);
  ASSERT_T_EQ(off_t, long);
#endif

#if defined (__3DS__)
  ASSERT_T_EQ(ino_t, uint32_t);
#elif defined (__sparc__)
  ASSERT_T_EQ(ino_t, unsigned long);
#else
  ASSERT_T_EQ(ino_t, unsigned short);
#endif

	ASSERT_T_EQ(fsblkcnt_t, uint64_t);
	ASSERT_T_EQ(fsfilcnt_t, uint32_t);
	ASSERT_T_EQ(id_t, uint32_t);

#if defined (__3DS__)
  ASSERT_T_EQ(key_t, int);
#else
  ASSERT_T_EQ(key_t, long);
#endif

#if !defined (__vita__)
	ASSERT_T_EQ(loff_t, long long);
#endif

#if defined (__3DS__)
  ASSERT_T_EQ(mode_t, unsigned int);
  ASSERT_T_EQ(nfds_t, uint32_t);
#else
  ASSERT_T_EQ(mode_t, uint32_t);
  ASSERT_T_EQ(nfds_t, unsigned int);
#endif

	ASSERT_T_EQ(nlink_t, unsigned short);

#if defined (__3DS__)
  ASSERT_T_EQ(pthread_t, unsigned long);
#else
  ASSERT_T_EQ(pthread_t, uint32_t);
#endif
	ASSERT_T_EQ(pthread_key_t, unsigned int);
	ASSERT_T_EQ(rlim_t, uint32_t);

#if defined (__3DS__)
  ASSERT_T_EQ(sa_family_t, uint16_t);
#else
  ASSERT_T_EQ(sa_family_t, uint8_t);
#endif

	ASSERT_T_EQ(socklen_t, uint32_t);
#if !defined (__vita__)
	ASSERT_T_EQ(speed_t, uint32_t);
#endif
	ASSERT_T_EQ(suseconds_t, int32_t);
#if !defined (__vita__)
	ASSERT_T_EQ(tcflag_t, unsigned int);
#endif
	ASSERT_T_EQ(useconds_t, uint32_t);

#if defined (__3DS__)
  ASSERT_T_EQ(time_t, long long);
#else
  ASSERT_T_EQ(time_t, int32_t);
#endif

  // The order of the `ai_addr` field in this struct is crucial
  // for converting between the Rust and C types.
  struct addrinfo_replica {
    int ai_flags;
    int ai_family;
    int ai_socktype;
    int ai_protocol;
    socklen_t ai_addrlen;

#if defined (ESP_PLATFORM)
    sockaddr* ai_addr;
#endif

    char* ai_canonname;

#if !defined (ESP_PLATFORM) // & !defined (PPC_NINTENDO or smth)
    struct sockaddr* ai_addr;
#endif

    struct addrinfo* ai_next;
  };
  ASSERT_T_EQ(struct addrinfo_replica, struct addrinfo);

  struct ip_mreq_replica {
    struct in_addr imr_multiaddr;
    struct in_addr imr_interface;
  };
  ASSERT_T_EQ(struct ip_mreq_replica, struct ip_mreq);

  struct linger_replica {
    int l_onoff;
    int l_linger;
  };
  ASSERT_T_EQ(struct linger_replica, struct linger);

#if !defined (__vita__)
  struct in_addr_replica {
    in_addr_t s_addr;
  };
  ASSERT_T_EQ(struct in_addr_replica, struct in_addr);
#endif

  struct hostent_replica {
    char* h_name;
    char** h_aliases;
    int h_addrtype;
    int h_length;
    char** h_addr_list;
    // char* h_addr;
  };
  ASSERT_T_EQ(struct hostent_replica, struct hostent);

  struct pollfd_replica {
    int fd;
    int events;
    int revents;
  };
  ASSERT_T_EQ(struct pollfd_replica, struct pollfd);

  struct lconv_replica {
    char* decimal_point;
    char* thousands_sep;
    char* grouping;
    char* int_curr_symbol;
    char* currency_symbol;
    char* mon_decimal_point;
    char* mon_thousands_sep;
    char* mon_grouping;
    char* positive_sign;
    char* negative_sign;
    char int_frac_digits;
    char frac_digits;
    char p_cs_precedes;
    char p_sep_by_space;
    char n_cs_precedes;
    char n_sep_by_space;
    char p_sign_posn;
    char n_sign_posn;
    char int_n_cs_precedes;
    char int_n_sep_by_space;
    char int_n_sign_posn;
    char int_p_cs_precedes;
    char int_p_sep_by_space;
    char int_p_sign_posn;
  };
  ASSERT_T_EQ(struct lconv_replica, struct lconv);

  struct tm_replica {
    int tm_sec;
    int tm_min;
    int tm_hour;
    int tm_mday;
    int tm_mon;
    int tm_year;
    int tm_wday;
    int tm_yday;
    int tm_isdst;
  };
  ASSERT_T_EQ(struct tm_replica, struct tm);

  struct statvfs_replica {
    unsigned long f_bsize;
    unsigned long f_frsize;
    fsblkcnt_t f_blocks;
    fsblkcnt_t f_bfree;
    fsblkcnt_t f_bavail;
    fsfilcnt_t f_files;
    fsfilcnt_t f_ffree;
    fsfilcnt_t f_favail;
    unsigned long f_fsid;
    unsigned long f_flag;
    unsigned long f_namemax;
  };
  ASSERT_T_EQ(struct statvfs_replica, statvfs);

  struct sigaction_replica {
    void* sa_handler;
    sigset_t sa_mask;
    int sa_flags;
  };
  ASSERT_T_EQ(struct sigaction_replica, sigaction);

  struct dirent_replica {
#if !defined (__vita__)
    ino_t d_ino;
    unsigned char d_type;
#else
    uint8_t __offset[88];
#endif
    char d_name[256];
  };
  ASSERT_T_EQ(struct dirent_replica, struct dirent);

  struct stack_t_replica {
    void* ss_sp;
    int ss_flags;
    size_t ss_size;
  };
  ASSERT_T_EQ(struct stack_t_replica, stack_t);

#define ULONG_SIZE 32
  ASSERT_EQ(ULONG_SIZE, sizeof (unsigned long) * 8);

  struct fd_set_replica { // Unverified
    unsigned long fds_bits[FD_SETSIZE / ULONG_SIZE];
  };
  ASSERT_T_EQ(struct fd_set_replica, fd_set);

  struct passwd_replica { // Unverified
    char* pw_name;
    char* pw_passwd;
    uid_t pw_uid;
    gid_t pw_gid;
    char* pw_gecos;
    char* pw_dir;
    char* pw_shell;
  };
  ASSERT_T_EQ(struct passwd_replica, struct passwd);

#define NCCS 32
#if !defined (__vita__)
  struct termios_replica { // Unverified
    tcflag_t c_iflag;
    tcflag_t c_oflag;
    tcflag_t c_cflag;
    tcflag_t c_lflag;
    cc_t c_line;
    cc_t c_cc[NCCS];
  };
  ASSERT_T_EQ(struct termios_replica, termios);
#endif

  struct sem_t_replica { // Unverified
    char __size[16];
  };
  ASSERT_T_EQ(struct sem_t_replica, sem_t);

#if !defined(__vita__)
  struct Dl_info_replica { // Unverified
    const char* dli_fname;
    void* dli_fbase;
    const char* dli_sname;
    void* dli_saddr;
  };
  ASSERT_T_EQ(struct Dl_info_replica, Dl_info);
#endif

  struct utsname_replica { // Unverified
    char sysname[65];
    char nodename[65];
    char release[65];
    char version[65];
    char machine[65];
    char domainname[65];
  };
  ASSERT_T_EQ(struct utsname_replica, struct utsname);

  struct cpu_set_t_replica { // Unverified
        uint32_t bits[32];
  };
  ASSERT_T_EQ(struct cpu_set_t_replica, cpu_set_t);

#if defined (ESP_PLATFORM)
#define __PTHREAD_INITIALIZER_BYTE 0xff
#define __SIZEOF_PTHREAD_ATTR_T 32
#define __SIZEOF_PTHREAD_MUTEX_T 4
#define __SIZEOF_PTHREAD_MUTEXATTR_T 12
#define __SIZEOF_PTHREAD_COND_T 4
#define __SIZEOF_PTHREAD_CONDATTR_T 8
#define __SIZEOF_PTHREAD_RWLOCK_T 4
#define __SIZEOF_PTHREAD_RWLOCKATTR_T 12
#define __SIZEOF_PTHREAD_BARRIER_T 32
#elif defined (__vita__)
#define __PTHREAD_INITIALIZER_BYTE 0xff
#define __SIZEOF_PTHREAD_ATTR_T 4
#define __SIZEOF_PTHREAD_MUTEX_T 4
#define __SIZEOF_PTHREAD_MUTEXATTR_T 4
#define __SIZEOF_PTHREAD_COND_T 4
#define __SIZEOF_PTHREAD_CONDATTR_T 4
#define __SIZEOF_PTHREAD_RWLOCK_T 4
#define __SIZEOF_PTHREAD_RWLOCKATTR_T 4
#define __SIZEOF_PTHREAD_BARRIER_T 4
#else
#define __PTHREAD_INITIALIZER_BYTE 0
#define __SIZEOF_PTHREAD_ATTR_T 56
#define __SIZEOF_PTHREAD_MUTEX_T 40
#define __SIZEOF_PTHREAD_MUTEXATTR_T 4
#define __SIZEOF_PTHREAD_COND_T 48
#define __SIZEOF_PTHREAD_CONDATTR_T 4
#define __SIZEOF_PTHREAD_RWLOCK_T 56
#define __SIZEOF_PTHREAD_RWLOCKATTR_T 8
#define __SIZEOF_PTHREAD_BARRIER_T 32
#endif

  struct pthread_mutex_t_replica { // Unverified
    uint8_t size[__SIZEOF_PTHREAD_MUTEX_T];
#if (POINTER_SIZE == 4) & (defined (__arm__) | defined (__PPC__) | defined (__mips__))
  } __attribute__ ((aligned (4)));
#else
  } __attribute__ ((aligned (8)));
#endif
  ASSERT_T_EQ(struct pthread_mutex_t_replica, pthread_mutex_t);

  struct pthread_rwlock_t_replica { // Unverified
    uint8_t size[__SIZEOF_PTHREAD_RWLOCK_T];
#if (POINTER_SIZE == 4) & (defined (__arm__) | defined (__PPC__) | defined (__mips__))
  } __attribute__ ((aligned (4)));
#else
  } __attribute__ ((aligned (8)));
#endif
  ASSERT_T_EQ(struct pthread_rwlock_t_replica, pthread_rwlock_t);

  struct pthread_mutexattr_t_replica { // Unverified
    uint8_t size[__SIZEOF_PTHREAD_MUTEXATTR_T];
#if (POINTER_SIZE == 4) | defined (__amd64__) | defined (__PPC64__) | defined (__mips__) | defined (__s390x__) | defined (__sparc__)
  } __attribute__ ((aligned (4)));
#else
  } __attribute__ ((aligned (8)));
#endif
  ASSERT_T_EQ(struct pthread_mutexattr_t_replica, pthread_mutexattr_t);

  struct pthread_cond_t_replica { // Unverified
    uint8_t size[__SIZEOF_PTHREAD_COND_T];
#if defined (__vita__)
  } __attribute__ ((aligned (4)));
#else
  } __attribute__ ((aligned (8)));
#endif
  ASSERT_T_EQ(struct pthread_cond_t_replica, pthread_cond_t);

  struct pthread_condattr_t_replica { // Unverified
    uint8_t size[__SIZEOF_PTHREAD_CONDATTR_T];
  } __attribute__ ((aligned (4)));
  ASSERT_T_EQ(struct pthread_condattr_t_replica, pthread_condattr_t);

  struct pthread_attr_t_replica { // Unverified
    uint8_t __size[__SIZEOF_PTHREAD_ATTR_T];
#if defined (__vita__)
  } __attribute__ ((aligned (4)));
#else
  };
#endif
  ASSERT_T_EQ(struct pthread_attr_t_replica, pthread_attr_t);

  struct pthread_rwlockattr_t_replica { // Unverified
    uint8_t __size[__SIZEOF_PTHREAD_RWLOCKATTR_T];
#if defined (__vita__)
  } __attribute__ ((aligned (4)));
#else
  };
#endif
  ASSERT_T_EQ(struct pthread_rwlockattr_t_replica, pthread_rwlockattr_t);

  // unverified constants
  pthread_mutex_t pthread_mutex_initializer = PTHREAD_MUTEX_INITIALIZER;
  struct pthread_mutex_t_replica pthread_mutex_initializer_replica;
  memset(&pthread_mutex_initializer_replica, __PTHREAD_INITIALIZER_BYTE, sizeof (pthread_mutex_initializer_replica));
  ASSERT_BYTES_EQ(&pthread_mutex_initializer, &pthread_mutex_initializer_replica);

  pthread_cond_t pthread_cond_initializer = PTHREAD_COND_INITIALIZER;
  struct pthread_cond_t_replica pthread_cond_initializer_replica;
  memset(&pthread_cond_initializer_replica, __PTHREAD_INITIALIZER_BYTE, sizeof (pthread_cond_initializer_replica));
  ASSERT_BYTES_EQ(&pthread_cond_initializer, &pthread_cond_initializer_replica);

  pthread_rwlock_t pthread_rwlock_initializer = PTHREAD_RWLOCK_INITIALIZER;
  struct pthread_rwlock_t_replica pthread_rwlock_initializer_replica;
  memset(&pthread_rwlock_initializer_replica, __PTHREAD_INITIALIZER_BYTE, sizeof (pthread_rwlock_initializer_replica));
  ASSERT_BYTES_EQ(&pthread_rwlock_initializer, &pthread_rwlock_initializer_replica);

#define __SIZEOF_PTHREAD_BARRIERATTR_T 4
#define __PTHREAD_MUTEX_HAVE_PREV 1
#define __PTHREAD_RWLOCK_INT_FLAGS_SHARED 1

	ASSERT_EQ(PTHREAD_MUTEX_NORMAL, 0);
	ASSERT_EQ(PTHREAD_MUTEX_RECURSIVE, 1);
	ASSERT_EQ(PTHREAD_MUTEX_ERRORCHECK, 2);
	#if defined(__3DS__) | defined (ESP_PLATFORM)
	ASSERT_EQ(FD_SETSIZE, 64);
	#elif defined(__vita__)
	ASSERT_EQ(FD_SETSIZE, 256);
	#else 
	ASSERT_EQ(FD_SETSIZE, 1024);
	#endif

	ASSERT_EQ(ENOENT, 2);
	ASSERT_EQ(ESRCH, 3);
	ASSERT_EQ(EINTR, 4);
	ASSERT_EQ(EIO, 5);
	ASSERT_EQ(ENXIO, 6);
	ASSERT_EQ(E2BIG, 7);
	ASSERT_EQ(ENOEXEC, 8);
	ASSERT_EQ(EBADF, 9);
	ASSERT_EQ(ECHILD, 10);
	ASSERT_EQ(EAGAIN, 11);
	ASSERT_EQ(ENOMEM, 12);
	ASSERT_EQ(EACCES, 13);
	ASSERT_EQ(EFAULT, 14);
	ASSERT_EQ(EBUSY, 16);
	ASSERT_EQ(EEXIST, 17);
	ASSERT_EQ(EXDEV, 18);
	ASSERT_EQ(ENODEV, 19);
	ASSERT_EQ(ENOTDIR, 20);
	ASSERT_EQ(EISDIR, 21);
	ASSERT_EQ(EINVAL, 22);
	ASSERT_EQ(ENFILE, 23);
	ASSERT_EQ(EMFILE, 24);
	ASSERT_EQ(ENOTTY, 25);
	ASSERT_EQ(ETXTBSY, 26);
	ASSERT_EQ(EFBIG, 27);
	ASSERT_EQ(ENOSPC, 28);
	ASSERT_EQ(ESPIPE, 29);
	ASSERT_EQ(EROFS, 30);
	ASSERT_EQ(EMLINK, 31);
	ASSERT_EQ(EPIPE, 32);
	ASSERT_EQ(EDOM, 33);
	ASSERT_EQ(ERANGE, 34);
	ASSERT_EQ(ENOMSG, 35);
	ASSERT_EQ(EIDRM, 36);
	ASSERT_EQ(EDEADLK, 45);
	ASSERT_EQ(ENOLCK, 46);
	ASSERT_EQ(ENOSTR, 60);
	ASSERT_EQ(ENODATA, 61);
	ASSERT_EQ(ETIME, 62);
	ASSERT_EQ(ENOSR, 63);
	ASSERT_EQ(ENOLINK, 67);
	ASSERT_EQ(EPROTO, 71);
	ASSERT_EQ(EMULTIHOP, 74);
	ASSERT_EQ(EBADMSG, 77);
	ASSERT_EQ(EFTYPE, 79);
	ASSERT_EQ(ENOSYS, 88);
	ASSERT_EQ(ENOTEMPTY, 90);
	ASSERT_EQ(ENAMETOOLONG, 91);
	ASSERT_EQ(ELOOP, 92);
	ASSERT_EQ(EOPNOTSUPP, 95);
	ASSERT_EQ(EPFNOSUPPORT, 96);
	ASSERT_EQ(ECONNRESET, 104);
	ASSERT_EQ(ENOBUFS, 105);
	ASSERT_EQ(EAFNOSUPPORT, 106);
	ASSERT_EQ(EPROTOTYPE, 107);
	ASSERT_EQ(ENOTSOCK, 108);
	ASSERT_EQ(ENOPROTOOPT, 109);
	ASSERT_EQ(ECONNREFUSED, 111);
	ASSERT_EQ(EADDRINUSE, 112);
	ASSERT_EQ(ECONNABORTED, 113);
	ASSERT_EQ(ENETUNREACH, 114);
	ASSERT_EQ(ENETDOWN, 115);
	ASSERT_EQ(ETIMEDOUT, 116);
	ASSERT_EQ(EHOSTDOWN, 117);
	ASSERT_EQ(EHOSTUNREACH, 118);
	ASSERT_EQ(EINPROGRESS, 119);
	ASSERT_EQ(EALREADY, 120);
	ASSERT_EQ(EDESTADDRREQ, 121);
	ASSERT_EQ(EMSGSIZE, 122);
	ASSERT_EQ(EPROTONOSUPPORT, 123);
	ASSERT_EQ(EADDRNOTAVAIL, 125);
	ASSERT_EQ(ENETRESET, 126);
	ASSERT_EQ(EISCONN, 127);
	ASSERT_EQ(ENOTCONN, 128);
	ASSERT_EQ(ETOOMANYREFS, 129);
	ASSERT_EQ(EDQUOT, 132);
	ASSERT_EQ(ESTALE, 133);
	ASSERT_EQ(ENOTSUP, 134);
	ASSERT_EQ(EILSEQ, 138);
	ASSERT_EQ(EOVERFLOW, 139);
	ASSERT_EQ(ECANCELED, 140);
	ASSERT_EQ(ENOTRECOVERABLE, 141);
	ASSERT_EQ(EOWNERDEAD, 142);
	ASSERT_EQ(EWOULDBLOCK, 11);

	ASSERT_EQ(F_DUPFD, 0);
	ASSERT_EQ(F_GETFD, 1);
	ASSERT_EQ(F_SETFD, 2);
	ASSERT_EQ(F_GETFL, 3);
	ASSERT_EQ(F_SETFL, 4);
	ASSERT_EQ(F_GETOWN, 5);
	ASSERT_EQ(F_SETOWN, 6);
	ASSERT_EQ(F_GETLK, 7);
	ASSERT_EQ(F_SETLK, 8);
	ASSERT_EQ(F_SETLKW, 9);
	ASSERT_EQ(F_RGETLK, 10);
	ASSERT_EQ(F_RSETLK, 11);
	ASSERT_EQ(F_CNVT, 12);
	ASSERT_EQ(F_RSETLKW, 13);
	ASSERT_EQ(F_DUPFD_CLOEXEC, 14);

	ASSERT_EQ(O_RDONLY, 0);
	ASSERT_EQ(O_WRONLY, 1);
	ASSERT_EQ(O_RDWR, 2);
	ASSERT_EQ(O_APPEND, 8);
	ASSERT_EQ(O_CREAT, 512);
	ASSERT_EQ(O_TRUNC, 1024);
	ASSERT_EQ(O_EXCL, 2048);
	ASSERT_EQ(O_SYNC, 8192);
	ASSERT_EQ(O_NONBLOCK, 16384);

	ASSERT_EQ(O_ACCMODE, 3);
	ASSERT_EQ(O_CLOEXEC, 0x80000);

	#if !defined (__3DS__)
	ASSERT_EQ(RTLD_LAZY, 0x1);
	#endif

	ASSERT_EQ(STDIN_FILENO, 0);
	ASSERT_EQ(STDOUT_FILENO, 1);
	ASSERT_EQ(STDERR_FILENO, 2);

	ASSERT_EQ(SEEK_SET, 0);
	ASSERT_EQ(SEEK_CUR, 1);
	ASSERT_EQ(SEEK_END, 2);

	#if !defined (__3DS__) & !defined (__vita__)
	ASSERT_EQ(FIOCLEX, 0x20006601);
	ASSERT_EQ(FIONCLEX, 0x20006602);
	#endif

	ASSERT_EQ(S_BLKSIZE, 1024);
	ASSERT_EQ(S_IREAD, 256);
	ASSERT_EQ(S_IWRITE, 128);
	ASSERT_EQ(S_IEXEC, 64);
	ASSERT_EQ(S_ENFMT, 1024);
	ASSERT_EQ(S_IFMT, 61440);
	ASSERT_EQ(S_IFDIR, 16384);
	ASSERT_EQ(S_IFCHR, 8192);
	ASSERT_EQ(S_IFBLK, 24576);
	ASSERT_EQ(S_IFREG, 32768);
	ASSERT_EQ(S_IFLNK, 40960);
	ASSERT_EQ(S_IFSOCK, 49152);
	ASSERT_EQ(S_IFIFO, 4096);
	ASSERT_EQ(S_IRUSR, 256);
	ASSERT_EQ(S_IWUSR, 128);
	ASSERT_EQ(S_IXUSR, 64);
	ASSERT_EQ(S_IRGRP, 32);
	ASSERT_EQ(S_IWGRP, 16);
	ASSERT_EQ(S_IXGRP, 8);
	ASSERT_EQ(S_IROTH, 4);
	ASSERT_EQ(S_IWOTH, 2);
	ASSERT_EQ(S_IXOTH, 1);

	#if !defined (__3DS__) & !defined (__vita__)
	ASSERT_EQ(SOL_TCP, 6);
	#endif

	ASSERT_EQ(PF_UNSPEC, 0);
	ASSERT_EQ(PF_INET, 2);
	ASSERT_EQ(PF_INET6, 23);

	ASSERT_EQ(AF_UNSPEC, 0);
	ASSERT_EQ(AF_INET, 2);

	ASSERT_EQ(CLOCK_REALTIME, 1);
	ASSERT_EQ(CLOCK_MONOTONIC, 4);
	#if !defined (__3DS__) & !defined (__vita__)
	ASSERT_EQ(CLOCK_BOOTTIME, 4); // 7
	#endif

	ASSERT_EQ(SOCK_STREAM, 1);
	ASSERT_EQ(SOCK_DGRAM, 2);

	ASSERT_EQ(SHUT_RD, 0);
	ASSERT_EQ(SHUT_WR, 1);
	ASSERT_EQ(SHUT_RDWR, 2);

	#if !defined (__3DS__) & !defined (__vita__)
	ASSERT_EQ(SO_BINTIME, 0x2000);
	ASSERT_EQ(SO_NO_OFFLOAD, 0x4000);
	ASSERT_EQ(SO_NO_DDP, 0x8000);
	ASSERT_EQ(SO_REUSEPORT_LB, 0x10000);
	ASSERT_EQ(SO_LABEL, 0x1009);
	ASSERT_EQ(SO_PEERLABEL, 0x1010);
	ASSERT_EQ(SO_LISTENQLIMIT, 0x1011);
	ASSERT_EQ(SO_LISTENQLEN, 0x1012);
	ASSERT_EQ(SO_LISTENINCQLEN, 0x1013);
	ASSERT_EQ(SO_SETFIB, 0x1014);
	ASSERT_EQ(SO_USER_COOKIE, 0x1015);
	ASSERT_EQ(SO_PROTOCOL, 0x1016);
	ASSERT_EQ(SO_PROTOTYPE, SO_PROTOCOL);
	ASSERT_EQ(SO_VENDOR, 0x80000000);
	#endif
	#if !defined (__3DS__)
	ASSERT_EQ(SO_DEBUG, 0x01);
	ASSERT_EQ(SO_ACCEPTCONN, 0x0002);
	#endif
	ASSERT_EQ(SO_REUSEADDR, 0x0004);
	#if !defined (__3DS__)
	ASSERT_EQ(SO_KEEPALIVE, 0x0008);
	ASSERT_EQ(SO_DONTROUTE, 0x0010);
	#endif
	ASSERT_EQ(SO_BROADCAST, 0x0020);
	#if !defined (__3DS__)
	ASSERT_EQ(SO_USELOOPBACK, 0x0040);
	#endif
	ASSERT_EQ(SO_LINGER, 0x0080);
	ASSERT_EQ(SO_OOBINLINE, 0x0100);
	#if !defined (__3DS__)
	ASSERT_EQ(SO_REUSEPORT, 0x0200);
	ASSERT_EQ(SO_TIMESTAMP, 0x0400);
	#endif
	#if !defined (__3DS__) & !defined (__vita__)
	ASSERT_EQ(SO_NOSIGPIPE, 0x0800);
	ASSERT_EQ(SO_ACCEPTFILTER, 0x1000);
	#endif
	ASSERT_EQ(SO_SNDBUF, 0x1001);
	ASSERT_EQ(SO_RCVBUF, 0x1002);
	ASSERT_EQ(SO_SNDLOWAT, 0x1003);
	ASSERT_EQ(SO_RCVLOWAT, 0x1004);
	#if !defined (__3DS__)
	ASSERT_EQ(SO_SNDTIMEO, 0x1005);
	ASSERT_EQ(SO_RCVTIMEO, 0x1006);
	#endif
	#if defined (__3DS__)
  ASSERT_EQ(SO_ERROR, 0x1009);
	#else
  ASSERT_EQ(SO_ERROR, 0x1007);
	#endif
	ASSERT_EQ(SO_TYPE, 0x1008);

	#if !defined (__3DS__) & !defined(__vita__)
	ASSERT_EQ(SOCK_CLOEXEC, O_CLOEXEC);
	#endif

	ASSERT_EQ(INET_ADDRSTRLEN, 16);

	#if !defined (__3DS__) & !defined (__vita__)
	ASSERT_EQ(IFF_UP, 0x1);
	ASSERT_EQ(IFF_BROADCAST, 0x2);
	ASSERT_EQ(IFF_DEBUG, 0x4);
	ASSERT_EQ(IFF_LOOPBACK, 0x8);
	ASSERT_EQ(IFF_POINTOPOINT, 0x10);
	ASSERT_EQ(IFF_NOTRAILERS, 0x20);
	ASSERT_EQ(IFF_RUNNING, 0x40);
	ASSERT_EQ(IFF_NOARP, 0x80);
	ASSERT_EQ(IFF_PROMISC, 0x100);
	ASSERT_EQ(IFF_ALLMULTI, 0x200);
	ASSERT_EQ(IFF_OACTIVE, 0x400);
	ASSERT_EQ(IFF_SIMPLEX, 0x800);
	ASSERT_EQ(IFF_LINK0, 0x1000);
	ASSERT_EQ(IFF_LINK1, 0x2000);
	ASSERT_EQ(IFF_LINK2, 0x4000);
	ASSERT_EQ(IFF_ALTPHYS, IFF_LINK2);
	ASSERT_EQ(IFF_MULTICAST, 0x8000);
	#endif

	ASSERT_EQ(TCP_NODELAY, 1);
	ASSERT_EQ(TCP_MAXSEG, 2);
	#if !defined (__3DS__) & !defined(__vita__)
	ASSERT_EQ(TCP_NOPUSH, 4);
	ASSERT_EQ(TCP_NOOPT, 8);
	ASSERT_EQ(TCP_KEEPIDLE, 256);
	ASSERT_EQ(TCP_KEEPINTVL, 512);
	ASSERT_EQ(TCP_KEEPCNT, 1024);
	#endif

	#if defined (__3DS__)
  ASSERT_EQ(IP_TOS, 7);
  ASSERT_EQ(IP_TTL, 8);
  ASSERT_EQ(IP_MULTICAST_LOOP, 9);
  ASSERT_EQ(IP_MULTICAST_TTL, 10);
  ASSERT_EQ(IP_ADD_MEMBERSHIP, 11);
  ASSERT_EQ(IP_DROP_MEMBERSHIP, 12);
  #else
  ASSERT_EQ(IP_TOS, 3);
  ASSERT_EQ(IP_TTL, 4);
  ASSERT_EQ(IP_MULTICAST_IF, 9);
  ASSERT_EQ(IP_MULTICAST_TTL, 10);
  ASSERT_EQ(IP_MULTICAST_LOOP, 11);
  ASSERT_EQ(IP_ADD_MEMBERSHIP, 12);
  ASSERT_EQ(IP_DROP_MEMBERSHIP, 13);
	#endif

	#if !defined (__3DS__) & !defined (__vita__)
	ASSERT_EQ(IPV6_UNICAST_HOPS, 4);
	ASSERT_EQ(IPV6_MULTICAST_IF, 9);
	ASSERT_EQ(IPV6_MULTICAST_HOPS, 10);
	ASSERT_EQ(IPV6_MULTICAST_LOOP, 11);
	ASSERT_EQ(IPV6_V6ONLY, 27);
	ASSERT_EQ(IPV6_JOIN_GROUP, 12);
	ASSERT_EQ(IPV6_LEAVE_GROUP, 13);
	ASSERT_EQ(IPV6_ADD_MEMBERSHIP, 12);
	ASSERT_EQ(IPV6_DROP_MEMBERSHIP, 13);
	#endif

	#if !defined (__vita__)
	ASSERT_EQ(HOST_NOT_FOUND, 1);
	ASSERT_EQ(NO_DATA, 2);
	ASSERT_EQ(NO_ADDRESS, 2);
	ASSERT_EQ(NO_RECOVERY, 3);
	ASSERT_EQ(TRY_AGAIN, 4);
	#endif

	ASSERT_EQ(AI_PASSIVE, 1);
	ASSERT_EQ(AI_CANONNAME, 2);
	ASSERT_EQ(AI_NUMERICHOST, 4);
	ASSERT_EQ(AI_NUMERICSERV, 0);
	ASSERT_EQ(AI_ADDRCONFIG, 0);

	ASSERT_EQ(NI_MAXHOST, 1025);
	ASSERT_EQ(NI_MAXSERV, 32);
	ASSERT_EQ(NI_NOFQDN, 1);
	ASSERT_EQ(NI_NUMERICHOST, 2);
	ASSERT_EQ(NI_NAMEREQD, 4);
	ASSERT_EQ(NI_NUMERICSERV, 0);
	ASSERT_EQ(NI_DGRAM, 0);

	#if !defined (__3DS__)
	ASSERT_EQ(EAI_AGAIN, 2);
	ASSERT_EQ(EAI_BADFLAGS, 3);
	ASSERT_EQ(EAI_FAIL, 4);
	#endif
	ASSERT_EQ(EAI_FAMILY, 5);
	ASSERT_EQ(EAI_MEMORY, 6);
	ASSERT_EQ(EAI_NONAME, 8);
	#if !defined (__3DS__)
	ASSERT_EQ(EAI_SERVICE, 9);
	#endif
	ASSERT_EQ(EAI_SOCKTYPE, 10);
	#if !defined (__3DS__)
	ASSERT_EQ(EAI_SYSTEM, 11);
	#endif
	#if !defined (__3DS__) & !defined (__vita__)
	ASSERT_EQ(EAI_BADHINTS, 12);
	ASSERT_EQ(EAI_PROTOCOL, 13);
	#endif
	#if !defined (__3DS__)
	ASSERT_EQ(EAI_OVERFLOW, 14);
	#endif

	ASSERT_EQ(EXIT_SUCCESS, 0);
	ASSERT_EQ(EXIT_FAILURE, 1);

	#if !defined (__3DS__)
	ASSERT_EQ(PRIO_PROCESS, 0);
	ASSERT_EQ(PRIO_PGRP, 1);
	ASSERT_EQ(PRIO_USER, 2);
	#endif
}

EDIT: I think i've modified the script code so some test maybe false positives/negatives on all/some platforms

@ivmarkov
Copy link
Contributor

ivmarkov commented Jan 5, 2024

Could anyone give me a summary of the discussion here? I'd like to help merge this but don't have enough context.

A lot of wrong definitions, particularly for newlib targets. The main reason, I would say, is that it has definitions from libctru for nintendo 3DS, which are super different from the most og newlib. This is because newlib code was introduced precisely for nintendo 3DS.

I no longer have time to untangle this problem to check field offests, struct alignments and sizes, etc. I would be happy to pass this torch to anyone interested. I thought of making some instrument to generate test C code, but it bacame too unwieldy to me. I also had hard time configuring ESP32 toolchain, it being a bash shell you have to dive into before getting access to compiler, so I haven't managed to check it either.

I can try to run your script on top of ESP IDF, but it would take a few days. Will report here once I have some progress.

@ivmarkov
Copy link
Contributor

ivmarkov commented Jan 5, 2024

Keep in mind horizon os of 3DS as to preserve these libctru definitions.

As long as all of these are marked with target_os = "horizon", sure. Otherwise how would you suggest we can figure out what is horizon-specific vs what is plain wrong. Or shall we treat all "wrong" definitions as horizon-specific?

@zetanumbers
Copy link
Author

zetanumbers commented Jan 6, 2024

Keep in mind horizon os of 3DS as to preserve these libctru definitions.

As long as all of these are marked with target_os = "horizon", sure. Otherwise how would you suggest we can figure out what is horizon-specific vs what is plain wrong. Or shall we treat all "wrong" definitions as horizon-specific?

Nothing you can't think of. I've roughtly translated newlib.rs into this script, but ideally there should be an automatic tool for every target.

EDIT: Sorry, didn't answer your question. For now wrong horizon os (or more correctly for exact armv6k-nintendo-3ds target) should probably stay the same.

@ian-h-chamberlain
Copy link
Contributor

Hi all, I meant to mention this sooner, but a long time ago I had been trying to add test support in libc-test/build.rs for the 3DS target, which might help avoid regressions here...

The changes were originally in Meziu#8 and are kinda hacky and not very portable since they require the 3ds toolchain to compile and a device/emulator to actually run the tests. I started trying to revive that test changes and so far have had a bunch of trouble compiling, but I can post back here if I get something working.

At the time, we didn't have a way to run tests automatically for the 3DS but have since created https://github.com/rust3ds/test-runner/. Maybe a potential path forward would be to add the 3DS to this repo's CI? It would probably take a fair amount of work to get set up, so I wouldn't want it to block this PR but just thinking about the long term

For now, this is probably the easiest option:

Or shall we treat all "wrong" definitions as horizon-specific?

I think this is basically what we did whenever we found a conflict between different definitions, and assumed that the 3DS definition was simply unusual. If it's not too much trouble to preserve all the current definitions with #[cfg(target_os = "horizon")] that's probably easiest, and in the long term we can go back and unify the definitions if they turn out to be wrong for the 3DS too (via the testing I mentioned above).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants