You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
According to the documentation, for adjust=False the formula for the recursive mean should be:
y_t = (1-alpha)y_{t-1} + alpha x_t
And for ignore_na=False the absolute positions should yield a factor of (1-alpha)^2 and alpha, which does not give the same answer as the invocation above:
btglau
changed the title
BUG: ewm with adjust=False and ignore_na_False does not seem to take NaN's into account
BUG: ewm with adjust=False and ignore_na=False does not seem properly take NaNs into account
May 2, 2024
I believe it's Nan causing an issue here. Shall I add an if statement to consider Nan as 0 or shall we throw an error here that Nan shouldn't be used to calculate in homogeneous_func() in rolling.py?
Also I was trying to enter in func(x, start, end, min_periods, *numba_args) in rolling.py to check how the values are calculated but could not get in this function? Any help in this regards.
btglau
changed the title
BUG: ewm with adjust=False and ignore_na=False does not seem properly take NaNs into account
BUG: ewm with adjust=False and ignore_na=False does not properly take NaNs into account
May 10, 2024
According to the documentation, if ignore_na=False then NaN should not be considered numerically, but should factor in for calculating the weights. It seems like your first proposal is in line with the documented behaviour.
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
According to the documentation, for
adjust=False
the formula for the recursive mean should be:y_t = (1-alpha)y_{t-1} + alpha x_t
And for
ignore_na=False
the absolute positions should yield a factor of(1-alpha)^2
andalpha
, which does not give the same answer as the invocation above:In contrast,
ignore_na=True
does give the right value:Expected Behavior
With
ignore_na=False
, the correct value in index 4 should be2.839506222222222
, not3.650794
.Installed Versions
INSTALLED VERSIONS
commit : bdc79c1
python : 3.11.5.final.0
python-bits : 64
OS : Darwin
OS-release : 23.4.0
Version : Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.2.1
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.3.1
Cython : 3.0.10
pytest : 7.4.0
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.20.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat : None
fastparquet : None
fsspec : 2023.10.0
gcsfs : None
matplotlib : 3.8.4
numba : 0.59.1
numexpr : 2.8.7
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 14.0.2
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : 2023.10.0
scipy : 1.13.0
sqlalchemy : 2.0.25
tables : 3.9.2
tabulate : 0.9.0
xarray : 2023.6.0
xlrd : 2.0.1
zstandard : 0.22.0
tzdata : 2023.3
qtpy : 2.4.1
pyqt5 : None
The text was updated successfully, but these errors were encountered: