IndexHierarchy.astype does strange things with boolean masks.

[ForeverWintr]

Description

Trying to astype levels of an IndexHierarchy doesn't seem to work correctly.

Example
Given this IH:

idx = sf.IndexHierarchy.from_labels([("1", 2), (3, "4"), ('5', '6')], name=(5, 6))
# <IndexHierarchy: (5, 6)>
# 1                        2
# 3                        4
# 5                        6
# <object>                 <object>

This looks like it should astype only the first level, but it astypes both.

idx.astype[[True, False]](str)
# <IndexHierarchy>
# 1                2
# 3                4
# 5                6
# <<U1>            <<U1>

This looks like it should astype both levels, but it astypes only the second

idx.astype[[True, True]](str)
# <IndexHierarchy>
# 1                2
# 3                4
# 5                6
# <object>         <<U1>

Platform

Run the following function (static-frame >= 0.8.1) and provide the results to define your platform and environment:

>>> import static_frame as sf
>>> sf.Platform.display()
<Series: platform>
<Index>
platform           Linux-5.13.0-48-generic-x86_64-with
sys.version        3.8.12 (default, Jun  8 2022, 11:51:38) [GCC 7.5.0]
static-frame       0.9.3
numpy              1.17.4
pandas             0.24.2
xlsxwriter         1.1.5
openpyxl           3.0.9
xarray             0.18.0
tables             3.6.1
pyarrow            0.17.0
msgpack            (1, 0, 0)
msgpack_numpy      None
<<U13>             <object>

[flexatone]

Many thanks for posting this question.

This behavior is correct and is consistent with other selection interfaces, but certainly seems surprising. Boolean selection in StaticFrame is only used if given a Boolean array. Here, we are using lists of bools, which is interpreted as any other selection list: as the "labels" to select. As IndexHierarchy selection is based on integer "iloc" positions, this can produce a confusing results, as every IndexHierarchy has positions 0 and 1.

We can see below that if we use a Boolean array instead of a list, we get the expected results.

>>> idx = sf.IndexHierarchy.from_labels([("1", 2), (3, "4"), ('5', '6')], name=(5, 6))                                                                                                              

>>> idx.astype[np.array((True, False))](str)                                                                                                                                                        
<IndexHierarchy>
1                2
3                4
5                6
<<U1>            <object>

>>> idx.astype[np.array((True, True))](str)                                                                                                                                                         
<IndexHierarchy>
1                2
3                4
5                6
<<U1>            <<U1>

>>> idx.astype[np.array((False, True))](str)                                                                                                                                                        
<IndexHierarchy>
1                2
3                4
5                6
<object>         <<U1>

[Acexxxxxxxxx]

@flexatone
It seems to work fine if we use 0,1 in a list to specify the iloc position.

 idx.astype[[0,1]](str)
<IndexHierarchy>
1                2
3                4
5                6
<<U1>            <<U1>

[flexatone]

@Acexxxxxxxxx : yes, that is expected. In that case we are selecting "columns" 0 and 1 (which is interpreted the same as False and True when given in a list).

[ForeverWintr]

Interesting! I wonder if it is worth having a special case for bools here, as it seems undesirable to treat them as integers in this case.

I think then that to change the dtypes of some levels in an index, e.g., convert objects to string, the best approach is to do this:

# 1. select types to convert
needs_conversion = idx.dtypes == np.dtype("O")

# 2. pass .values to astype:
converted=idx.astype[needs_conversion.values](str)

[flexatone]

@ForeverWintr , thanks for your comments.

Can you elaborate on what you suggest regarding bools? I think I misspoke when I said that (on a depth-2 index) np.array([False, True]) will be treated the same as [0, 1]; np.array([False, True]) would be the same as simply [1].

Regarding the "best approach", the following works exactly as you suggest:

>>> idx = sf.IndexHierarchy.from_product(('a', 'b'), (1, 2), (True, False))
>>> idx
<IndexHierarchy>
a                1       True
a                1       False
a                2       True
a                2       False
b                1       True
b                1       False
b                2       True
b                2       False
<<U1>            <int64> <bool>

>>> idx.astype[(idx.dtypes == bool).values](str)
<IndexHierarchy>
a                1       True
a                1       False
a                2       True
a                2       False
b                1       True
b                1       False
b                2       True
b                2       False
<<U1>            <int64> <<U5>

[ForeverWintr]

Your example shows the same thing as mine, right?

If I understand what's happening, astype[np.array([False, True])] is treated the same as simply, astype[[1]], while astype[[False, True]] is treated the same as astype[[0, 1]]. In fact astype[[False, True, True, True, True]] is also treated the same as astype[[0, 1]]!

Re-reading this, I'm not sure if I understood your original comment correctly.

Boolean selection in StaticFrame is only used if given a Boolean array.

I don't think this is true, as there are cases where a boolean list works as (I) expected. For example, in loc selection:

f = sf.Frame.from_records([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
f.loc[[True, False, True]]
<Frame>
<Index> 0       1       2       <int64>
<Index>
0       1       2       3
2       7       8       9
<int64> <int64> <int64> <int64>

I see this is not the case in all selections though.

I guess then my suggestion was that static frame treat boolean lists the same as boolean arrays, but at the time I though astype was the only interface that didn't do that. I see now that that's not the case.

However, selection with a boolean list works in numpy; should it work in more cases in static frame too?

[flexatone]

Thanks for the example of a list of bool working as an Boolean np.array... that is not expected and I will investigate (#497)

With NumPy, selection with a list of bool is never ambiguous as data is only labeled by integers. With StaticFrame, data can be labelled by any hashable, including Booleans, leading to some potentially very ambiguous selections if we do not exclusively treat Boolean np.array as Boolean selectors. The following contrived example makes this explicit, but there are other, more subtle challenges.

>>> f = sf.Frame.from_fields((list('abc'), list('def')), columns=(True, False))                                                                                                                                                                                         
>>> f                                                                                                                                                                                                                                                                   
<Frame>
<Index> True  False <bool>
<Index>
0       a     d
1       b     e
2       c     f
<int64> <<U1> <<U1>

>>> f[[False, True]]  # a selection list                                                                                                                                                                                                                                                   
<Frame>
<Index> False True  <bool>
<Index>
0       d     a
1       e     b
2       f     c
<int64> <<U1> <<U1>

>>> f[np.array([False, True])]   # a Boolean selection                                                                                                                                                                                                                                  
<Frame>
<Index> False <bool>
<Index>
0       d
1       e
2       f
<int64> <<U1>

Putting the responsibility of the caller to make explicit what type of selection they are doing by forcing the usage of Boolean arrays is well within the spirit of StaticFrame's stricter interfaces, I believe.

[ForeverWintr]

Good point about the possibility of Boolean labels!

Is this the only case where selecting with an array changes the selection behavior? It seems to me that arrays are treated the same as lists for other selections, e.g., f[['a', 'b']] is equivalent to f[np.array(['a', 'b'])].

[flexatone]

Yes, I think that is correct: in other situations you can use a list or an array to do selections.