Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error with more than 2^31-1 observations #43

Open
mcaceresb opened this issue Aug 19, 2018 · 5 comments
Open

Error with more than 2^31-1 observations #43

mcaceresb opened this issue Aug 19, 2018 · 5 comments
Labels
wontfix

Comments

@mcaceresb
Copy link
Owner

A bug in Stata causes gtools to exit with error if the user has more than 2^31-1 observations in memory. See this bug report.

I contacted StataCorp about it and they replied:

The SPI can work with datasets containing up to 2^31-1 observations. Our development group is looking into modifying future versions of the SPI to allow more observations.
mcaceresb added a commit that referenced this issue Aug 19, 2018
@mcaceresb
Merge 'develop' for gtools-1.0.3; exit if too many obs: SPI 32-bit in…
1714770 
…teger bug

* Closes #43
* Added option to selectively test via gtools, test[]
* Gtools exits with error if `_N > 2^31-1` and points the user to the
  pertinent bug report. The SPI uses long integers (32-bit) instead of
  long long integers (64-bit) for its observation numbers (not to
  mention theyare signed integers, but alas). At any rate, this de facto
  limits the number of observations to 2^31-1.
@mcaceresb
Copy link
Owner Author

Re-open because technically it hasn't been fixed. It just throws the correct error now.

@mcaceresb mcaceresb reopened this Aug 21, 2018
@miserton
Copy link

miserton commented Aug 9, 2022

Any update on this bug - is it correctable now?

@mcaceresb
Copy link
Owner Author

@miserton Not afaik. I would contact StataCorp to ask about updates to this point, since I cannot fix it until they update the SPI. Sorry!

@miserton
Copy link

I reached out to StataCorp:

The stplugin.h has been frozen so that we do not make any changes that could break commands that are still using this code. For 64-bit integer support we recommend working with Java or Python code instead of the older 32-bit C plugin limits.

Unless gtools could call the C code through Java or Python, it doesn't look like the 2.1B limitation will be overcome.

@mcaceresb
Copy link
Owner Author

@miserton That's a bit of a strange answer since changing stplugin.h in particular would only involve adding a long long type (and in my system long is 64-bit anyway). I always thought the issue were the internals, and that chancing the internal function to take 64-bit input was difficult for some reason. But I don't really know.

In any case, you are right this is unlikely to be resolved any time soon, if ever.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
wontfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mcaceresb @miserton