Skip to content

Error in openFeather(path) : Invalid: Not a feather file #315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yannikschaelte opened this issue Jun 9, 2020 · 7 comments
Closed

Error in openFeather(path) : Invalid: Not a feather file #315

yannikschaelte opened this issue Jun 9, 2020 · 7 comments
Labels

Comments

@yannikschaelte
Copy link
Member

When exporting the database to feather format, e.g. via

$ abc-export --db test.db --out test.feather --format feather --generation all

and importing in R via

> library('feather')
> data = read_feather('/tmp/test.feather')

we get

Error in openFeather(path) : Invalid: Not a feather file

this happens only for some versions, e.g. pyabc 0.10.3, pandas 1.0.4, R 3.6.3.

@yannikschaelte
Copy link
Member Author

This error can be reproduced quite simply:

import pandas as pd
df = pd.DataFrame({'a': [1,2,3], 'b': [4,5,6]})
df.to_feather('out.f')

same error

@yannikschaelte
Copy link
Member Author

See also pandas-dev/pandas#34670

@yannikschaelte
Copy link
Member Author

Closing here as this is not a problem of pyabc. One should use the latest arrow packages (or feather, at some point) on python and R side, see the above link.

@nbungi
Copy link
Contributor

nbungi commented Dec 2, 2020

Here is a work-around, as I encounter this issue even though I am using the latest arrow(2.0.0)/feather(0.3.5) packages in R.
Using pyarrow(2.0.0) one can read and rewrite the feather file in "version 1", which is then readable in R.
import pyarrow.feather as feather
feather_db = feather.read_feather('my_db.feather')
feather.write_feather(feather_db, 'my_db_v1.feather', version=1)

@chunjie-sam-liu
Copy link

Thanks to @nbungi, I find this link https://arrow.apache.org/docs/python/feather.html.

Writing Version 1 (V1) Files
For compatibility with libraries without support for Version 2 files, you can write the version 1 format by passing version=1 to write_feather. We intend to maintain read support for V1 for the foreseeable future.

@JohannesWiesner
Copy link

For me, when using version=1 and I try to load the .feather file in R-Studio, RStudio crashes.

@yannikschaelte
Copy link
Member Author

The feather format appears to be somewhat cumbersome, with different tools implementing/supporting different versions etc.. Not sure what lead to this error, but just to recapitulate a few options:

  • As pointed out above by @nbungi , you can convert to version 1 post-hoc. Alternatively, you can also programmatically extract the database in https://github.com/ICB-DCM/pyABC/blob/main/pyabc/storage/export.py#L83 (this is the function called by abc-export) and then apply pandas.to_feather(filename, version=1).
  • When importing in R, at least either feather::read_feather or arrow::read_feather can be used, with the latter in particular for version 1 exports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants