ๅๆ๏ผhttps://datascienceatthecommandline.com/2e/chapter-3-obtaining-data.html
่ดก็ฎ่ ๏ผTing-xin
ๆฌ็ซ ่ฎจ่ฎบ OSEMN ๆจกๅ็็ฌฌไธๆญฅ๏ผ่ทๅๆฐๆฎใๆฏ็ซ๏ผๆฒกๆไปปไฝๆฐๆฎ๏ผๆไปฌๅฐฑๆฒกๆๅคๅฐๆฐๆฎ็งๅญฆๅฏไปฅๅใๆๅ่ฎพไฝ ๅทฒ็ปๆไบ่งฃๅณๆฐๆฎ็งๅญฆ้ฎ้ขๆ้็ๆฐๆฎ๏ผ็ฌฌไธๆญฅไฝ ้่ฆๆ่ฟไบๆฐๆฎๆพๅฐไฝ ็็ต่ไธ๏ผไนๅฏ่ฝๆพๅฐ Docker ๅฎนๅจ้๏ผใ
ๅจ Unix ๅฒๅญฆไธญ๏ผๆๆฌๆฏไธไธช้็จๆฅๅฃใๅ ไนๆฏไธชๅฝไปค่กๅทฅๅ ท้ฝๅฐๆๆฌไฝไธบ่พๅ ฅ๏ผๆ่ ไปฅๆๆฌไฝไธบ่พๅบ๏ผๆ่ ไธค่ ้ฝๆใ่ฟๅฐฑๆฏไธบไปไนๅฝไปค่กๅทฅๅ ทๅฏไปฅๅพๅฅฝๅฐๅๅๅทฅไฝ็ไธป่ฆๅๅ ใ็ถ่๏ผๆญฃๅฆๆไปฌๅฐ็ๅฐ็๏ผๅณไฝฟๅชๆฏๆๆฌไนๅฏไปฅๆๅค็งๅฝขๅผใ
ๆไปฌๅฏไปฅ้่ฟๅค็งๆนๅผ่ทๅๆฐๆฎ๏ผไพๅฆไปๆๅกๅจไธ่ฝฝๆฐๆฎใๆฅ่ฏขๆฐๆฎๅบๆ่ฟๆฅๅฐ Web APIใๆๆถ๏ผๆฐๆฎไปฅๅ็ผฉ็ๅฝขๅผๆไบ่ฟๅถๆ ผๅผๅบ็ฐ๏ผๅฆ Microsoft Excel ็ตๅญ่กจๆ ผใๅจ่ฟไธ็ซ ไธญ๏ผๆไปฌๅฐ่ฎจ่ฎบไบๅ ไธชๆๅฉไบไปๅฝไปค่ก่งฃๅณ่ฟไธช้ฎ้ข็ๅทฅๅ
ท๏ผๅ
ๆฌ๏ผcurl๏ผin2csv๏ผsql2csv๏ผไปฅๅtarใ
ๅจๆฌ็ซ ไธญ๏ผไฝ ๅฐๅญฆไน ๅฆไฝ๏ผ
- ๅฐๆฌๅฐๆไปถๅคๅถๅฐ Docker ้ๅ
- ไปไบ่็ฝไธ่ฝฝๆฐๆฎ
- ่งฃๅ็ผฉๆไปถ
- ไป็ตๅญ่กจๆ ผไธญๆๅๆฐๆฎ
- ๆฅ่ฏขๅ ณ็ณปๆฐๆฎๅบ
- ่ฐ็จ Web API
้ฆๅ ๆๅผ็ฌฌไธ็ซ ็็ฎๅฝ๏ผ
$ cd /data/ch03
$ l
total 924K
-rw-r--r-- 1 dst dst 627K Mar 3 10:41 logs.tar.gz
-rw-r--r-- 1 dst dst 189K Mar 3 10:41 r-datasets.db
-rw-r--r-- 1 dst dst 149 Mar 3 10:41 tmnt-basic.csv
-rw-r--r-- 1 dst dst 148 Mar 3 10:41 tmnt-missing-newline.csv
-rw-r--r-- 1 dst dst 181 Mar 3 10:41 tmnt-with-header.csv
-rw-r--r-- 1 dst dst 91K Mar 3 10:41 top2000.xlsx่ทๅ่ฟไบๆไปถๅทฒ็ปๅจ็ฌฌไบ็ซ ไธญๅ่ฟไบใไปปไฝๅ ถไปๆไปถ้ฝๆฏไฝฟ็จๅฝไปค่กๅทฅๅ ทไธ่ฝฝๆ็ๆ็ใ
ไธ็งๅธธ่ง็ๆ ๅตๆฏ๏ผไฝ ่ชๅทฑ็่ฎก็ฎๆบไธๅทฒ็ปๆไบ้่ฆ็ๆไปถ๏ผๆฌ่ไป็ปไบๅฆไฝๅฐ่ฟไบๆไปถๆพๅ ฅ Docker ๅฎนๅจใ
ๆๅจ็ฌฌไบ็ซ ๆๅฐ Docker ๅฎนๅจๆฏไธไธช้็ฆป็่ๆ็ฏๅขใไฝๆฏๆไธไธชไพๅค๏ผๆไปถๅฏไปฅๅจ Docker ๅฎนๅจไธญ่ฟๅบไผ ่พใ่ฟ่กdocker run็ๆฌๅฐ็ฎๅฝไผ่ขซๆ ๅฐๅฐ Docker ๅฎนๅจไธญ็ไธไธช็ฎๅฝใ่ฟไธช็ฎๅฝๅซๅ/dataใๆณจๆ่ฟไธๆฏไธป็ฎๅฝ๏ผไธป็ฎๅฝๆฏ/home/dstใ
ๅฆๆไฝ ็ๆฌๅฐ่ฎก็ฎๆบไธๆไธไธชๆๅคไธชๆไปถ๏ผๅนถไธไฝ ๆณๅฏนๅฎไปฌๅบ็จไธไบๅฝไปค่กๅทฅๅ
ท๏ผ้ฃไนไฝ ้่ฆๅฐ่ฟไบๆไปถๅคๅถๆ็งปๅจๅฐ้ฃไธชๆ ๅฐ็็ฎๅฝไธญใๅ่ฎพไฝ ็ไธ่ฝฝ็ฎๅฝไธญๆไธไธชๅไธบlogs.csv็ๆไปถ๏ผ็ฐๅจๆไปฌๆฅๅคๅถๆไปถใ
ๅฆๆไฝ ๆญฃๅจ่ฟ่ก Windows๏ผ่ฏทๆๅผๅฝไปคๆ็คบ็ฌฆๆ PowerShell ๅนถ่ฟ่กไปฅไธไธคไธชๅฝไปค๏ผ
> cd %UserProfile%\Downloads
> copy logs.csv MyDataScienceToolbox\ๅฆๆไฝ ่ฟ่ก็ๆฏ Linux ๆ macOS๏ผ่ฏทๆๅผไธไธช็ป็ซฏๅนถๅจไฝ ็ๆไฝ็ณป็ปไธๆง่กไปฅไธๅฝไปค๏ผ่ไธๆฏๅจ Docker ๅฎนๅจไธญ๏ผ:
$ cp ~/Downloads/logs.csv ~/my-data-science-toolboxไฝ ไนๅฏไปฅไฝฟ็จๅพๅฝขๆไปถ็ฎก็ๅจ๏ผๅฆ Windows Explorer ๆ macOS Finder)ๅฐๆไปถๆๆพๅฐๆญฃ็กฎ็็ฎๅฝไธญใ
ๆฏซๆ ็้ฎ๏ผไบ่็ฝๅทฒ็ปๆไธบไบๆฐๆฎ็ๆๅคงๆฅๆบใๅฝไปไบ่็ฝไธ่ฝฝๆฐๆฎๆถ๏ผๅฝไปค่กๅทฅๅ
ทcurl่ขซ่ฎคไธบๆฏๅฝไปค่กไธญ็็ๅฃซๅๅใ
ๅฝไฝ ๆต่งๅฐไธไธชไปฃ่กจ็ปไธ่ตๆบๅฎไฝ็ฌฆ็ URL ๆถ๏ผไฝ ็ๆต่งๅจไผๆธฒๆๅฎไธ่ฝฝ็ๆฐๆฎใไพๅฆ๏ผๆต่งๅจไผๅ็ฐ HTML ๆไปถ๏ผ่ชๅจๆญๆพ่ง้ขๆไปถ๏ผๆพ็คบ PDF ๆไปถใ็ถ่๏ผๅฝไฝ ไฝฟ็จcurlๆฅ่ฎฟ้ฎไธไธช URL ๆถ๏ผๅฎไผไธ่ฝฝๆฐๆฎ๏ผๅนถๅจ้ป่ฎคๆ
ๅตไธๅฐๅ
ถๆๅฐๅบๆฅใcurlไธไผๅไปปไฝ่งฃ้ๅๆธฒๆ๏ผไฝๅนธ่ฟ็ๆฏๅฏไปฅไฝฟ็จๅ
ถไปๅฝไปค่กๅทฅๅ
ทๆฅ่ฟไธๆญฅๅค็ๆฐๆฎใ
ๆ็ฎๅ็่ฐ็จcurlๆฏๆๅฎไธไธช URL ไฝไธบๅฝไปค่กๅๆฐใ็ฐๅจ่ฎฉๆไปฌ่ฏ็ไป็ปดๅบ็พ็งไธ่ฝฝไธ็ฏๆ็ซ ๏ผ
$ curl "https://en.wikipedia.org/wiki/List_of_windmills_in_the_Netherlands" |
> trim # โ
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<!
DOCTYPE html>
<html class="client-nojs" lang="en" dir="ltr">
<head>
<meta charset="UTF-8"/>
<title>List of windmills in the Netherlands - Wikipedia</title>
<script>document.documentElement.className="client-js";RLCONF={"wgBreakFrames":โฆ
"wgRelevantPageName":"List_of_windmills_in_the_Netherlands","wgRelevantArticleIโฆ
,"site.styles":"ready","user.styles":"ready","ext.globalCssJs.user":"ready","usโฆ
"ext.growthExperiments.SuggestedEditSession"];</script>
<script>(RLQ=window.RLQ||[]).push(function(){mw.loader.implement("user.options@โฆ
100 250k 100 250k 0 0 866k 0 --:--:-- --:--:-- --:--:-- 891k
โฆ with 1760 more linesโ ่ฎฐไฝ๏ผtrimๅชๆฏ็จๆฅ่ฎฉ่พๅบๅพๅฅฝๅฐ้ๅบ็ป็ซฏ
ๅฆไฝ ๆ่ง๏ผcurlไธ่ฝฝ็ปดๅบ็พ็ง็ๆๅกๅจ่ฟๅ็ๅๅง HTML ไธ่ฟ่กไปปไฝ่งฃ้๏ผๆๆๅ
ๅฎน็ซๅณๆๅฐๅจๆ ๅ่พๅบไธใๅ ไธบ่ฟไธช URL๏ผไฝ ไผ่ฎคไธบ่ฟ็ฏๆ็ซ ไผๅๅบ่ทๅ
ฐๆๆ็้ฃ่ฝฆใ็ถ่๏ผๆพ็ถๆๅคชๅค็้ฃ่ฝฆ็ไธๆฅ๏ผๆฏไธช็้ฝๆ่ชๅทฑ็็ฝ้กตใไปคไบบ็่ฟทใ
้ป่ฎคๆ
ๅตไธ๏ผcurlไผ่พๅบไธไธช่ฟๅบฆๆก๏ผๆพ็คบไธ่ฝฝ้ๅบฆๅ้ขๆๅฎๆๆถ้ดใ่ฟไธช่พๅบไธๆฏๅๅฐๆ ๅ่พๅบ๏ผ่ๆฏไธไธชๅ็ฌ็้้๏ผ็งฐไธบๆ ๅ้่ฏฏ๏ผๆไปฅๅฝไฝ ๅจ็ฎก้ไธญๆทปๅ ๅฆไธไธชๅทฅๅ
ทๆถ๏ผๅฎไปฌไน้ดไธไผๅนฒๆฐใ่ฝ็ถ่ฟไบไฟกๆฏๅจไธ่ฝฝ้ๅธธๅคง็ๆไปถๆถไผๅพๆ็จ๏ผไฝๅฎ้ๅธธไผๅๆฃๆไปฌ็ๆณจๆๅ๏ผๆไปฅๆๆๅฎไบ-s้้กนๆฅ่ฎฉๅฟฝ็ฅ่ฟไธช่พๅบใ
$ curl -s "https://en.wikipedia.org/wiki/List_of_windmills_in_Friesland" |
> pup -n 'table.wikitable tr' # โ
234โ ๆไผ่ฎจ่ฎบpup๏ผไธไธชๆนไพฟ็ๆๅ็ฝ็ซ็ๅทฅๅ
ท๏ผๆด่ฏฆ็ป็ๅ
ๅฎนๅจ็ฌฌไบ็ซ ใ
ไฝ ็ฅ้ๅ๏ผไป ๅจๅผ้ๆฏๅ ฐ็ๅฐฑๆ 234 ๅบง้ฃ่ฝฆ๏ผ๏ผ่ฏ่ ไนไธๆไธบๅฅ็ช็ถๆฅ่ฟไนไธๅฅ
ไฝ ๅฏไปฅ้่ฟๆทปๅ -O้้กนๅฐcurl็่พๅบไฟๅญๅฐๆไปถไธญ๏ผๆไปถๅๅฐๅบไบ URL ็ๆๅไธ้จๅใ
$ curl -s "https://en.wikipedia.org/wiki/List_of_windmills_in_Friesland" -O
$ l
total 1.4M
-rw-r--r-- 1 dst dst 432K Mar 3 10:41 List_of_windmills_in_Friesland
-rw-r--r-- 1 dst dst 627K Mar 3 10:41 logs.tar.gz
-rw-r--r-- 1 dst dst 189K Mar 3 10:41 r-datasets.db
-rw-r--r-- 1 dst dst 149 Mar 3 10:41 tmnt-basic.csv
-rw-r--r-- 1 dst dst 148 Mar 3 10:41 tmnt-missing-newline.csv
-rw-r--r-- 1 dst dst 181 Mar 3 10:41 tmnt-with-header.csv
-rw-r--r-- 1 dst dst 91K Mar 3 10:41 top2000.xlsxๅฆๆไฝ ไธๅๆฌข่ฟไธชๆไปถๅ๏ผ้ฃไนไฝ ๅฏไปฅ้ๆฉไฝฟ็จ-o้้กนๅไธไธชๆไปถๅๆฅไฟๅญๆไปถ๏ผๆ่
ๅฐ่พๅบ้ๅฎๅๅฐไธไธชๆฐ็ๆไปถ๏ผ
$ curl -s "https://en.wikipedia.org/wiki/List_of_windmills_in_Friesland" > fries
land.htmlcurlๆปๅ
ฑๆฏๆ 20 ๅค็งๅ่ฎฎใไป FTP ๆๅกๅจ๏ผไปฃ่กจๆไปถไผ ่พๅ่ฎฎ๏ผไธ่ฝฝๆไปถๅๆ ทไนๅฏไปฅไฝฟ็จcurl๏ผไธ้ขๆพ็คบไปftp.gnu.orgไธ่ฝฝๆไปถwelcome.msg:
$ curl -s "ftp://ftp.gnu.org/welcome.msg" | trim
NOTICE (Updated October 15 2021):
If you maintain scripts used to access ftp.gnu.org over FTP,
we strongly encourage you to change them to use HTTPS instead.
Eventually we hope to shut down FTP protocol access, but plan
to give notice here and other places for several months ahead
of time.
--
โฆ with 19 more linesๅฆๆๆๅฎ็ URL ๆฏไธไธช DICT ๅ่ฎฎ๏ผcurlๅฐๅๅบ่ฏฅ็ฎๅฝ็ๅ
ๅฎนใๅฝ URL ๅๅฏ็ ไฟๆคๆถ๏ผไฝ ๅฏไปฅไฝฟ็จ-u้้กนๆๅฎ็จๆทๅๅๅฏ็ ใDICT ๅ่ฎฎๅ
่ฎธไฝ ่ฎฟ้ฎๅ็งๅญๅ
ธๅๆฅๆพๅฎไน๏ผๆ นๆฎใๅฝ้
่ฑ่ฏญๅไฝ่ฏๅ
ธใ๏ผไธ้ขๆฏโ้ฃ่ฝฆโ็ๅฎไน๏ผ
$ curl -s "dict://dict.org/d:windmill" | trim
220 dict.dict.org dictd 1.12.1/rf on Linux 4.19.0-10-amd64 <auth.mime> <9813708โฆ
250 ok
150 1 definitions retrieved
151 "Windmill" gcide "The Collaborative International Dictionary of English v.0โฆ
Windmill \Wind"mill`\, n.
A mill operated by the power of the wind, usually by the
action of the wind upon oblique vanes or sails which radiate
from a horizontal shaft. --Chaucer.
[1913 Webster]
.
โฆ with 2 more lines็ถ่๏ผๅฝไปไบ่็ฝไธ่ฝฝๆฐๆฎๆถ๏ผๅ่ฎฎๅพๅฏ่ฝๆฏ HTTP๏ผๅ ๆญค URL ๅฐไปฅhttp://ๆhttps://ๅผๅคดใ
ๅฝไฝ ่ฎฟ้ฎไธไธช็ผฉ็ญ็็ฝๅๆถ๏ผๆฏๅฆไปฅhttp://bit.ly/ๆhttp://t.co/ๅผๅคด็็ฝๅ๏ผไฝ ็ๆต่งๅจไผ่ชๅจๅฐไฝ ้ๅฎๅๅฐๆญฃ็กฎ็ไฝ็ฝฎใ็ถ่๏ผไฝฟ็จcurl๏ผไฝ ้่ฆๆๅฎ-Lๆ--location้้กนไปฅไพฟ่ขซ้ๅฎๅใๅฆๆๆฒกๆ๏ผไฝ ๅฏ่ฝไผๅพๅฐ่ฟๆ ท็็ปๆ๏ผ
$ curl -s "https://bit.ly/2XBxvwK"
<html>
<head><title>Bitly</title></head>
<body><a href="https://youtu.be/dQw4w9WgXcQ">moved here</a></body>
</html>%ๆๆถๅไฝ ไปไนไนๅพไธๅฐ๏ผๅฐฑๅๆไปฌๆ็ งไธ้ขๆๅฐ็็ฝๅ๏ผ
$ curl -s "https://youtu.be/dQw4w9WgXcQ"้่ฟๆๅฎ-Iๆ--head้้กน๏ผcurlๅช่ทๅๅๅบ็ HTTP ๅคด๏ผ่ฟๅ
่ฎธไฝ ๆฃๆฅๆๅกๅจ่ฟๅ็็ถๆไปฃ็ ๅๅ
ถไปไฟกๆฏใ
$ curl -sI "https://youtu.be/dQw4w9WgXcQ" | trim
HTTP/2 303
content-type: application/binary
x-content-type-options: nosniff
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Thu, 03 Mar 2022 09:41:56 GMT
location: https://www.youtube.com/watch?v=dQw4w9WgXcQ&feature=youtu.be
content-length: 0
x-frame-options: SAMEORIGIN
โฆ with 11 more linesไฟกๆฏไธญ็ฌฌไธ่กๆพ็คบไบ HTTP ๅ่ฎฎๅ็ถๆ็ ๏ผๅจๆฌไพไธญๆฏ 303ใไฝ ่ฟๅฏไปฅ็ๅฐ่ฏฅ URL ้ๅฎๅๅฐ็ไฝ็ฝฎใๅฆๆcurlๆฒกๆ็ปไฝ ้ขๆ็็ปๆ๏ผๆฃๆฅๆ ้ขๅนถ่ทๅพ็ถๆ็ ๆฏไธไธชๆ็จ็่ฐ่ฏๅทฅๅ
ทใๅ
ถไปๅธธ่ง็ HTTP ็ถๆไปฃ็ ๅ
ๆฌ 404๏ผๆชๆพๅฐ๏ผๅ 403๏ผ็ฆๆญข๏ผใ็ปดๅบ็พ็งๅๅบไบๆๆ HTTP ็ถๆ็ ใ
ๆปไน๏ผcurlๆฏไธไธชๆ็จ็ไปไบ่็ฝไธ่ฝฝๆฐๆฎ็ๅฝไปค่กๅทฅๅ
ทใๅฎ็ไธไธชๆๅธธ่ง็้้กนๆฏ-sๅฟฝ็ฅ่ฟๅบฆๆกใ-uๆๅฎ็จๆทๅๅๅฏ็ ใ-L่ชๅจ่ท้้ๅฎๅใ่ฏทๅ้
ๅ
ถๆๅ้กตไบ่งฃๆดๅคไฟกๆฏ๏ผ
$ man curl | trim 20
curl(1) Curl Manual curl(1)
NAME
curl - transfer a URL
SYNOPSIS
curl [options / URLs]
DESCRIPTION
curl is a tool to transfer data from or to a server, using one of the
supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP,
IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP,
SMB, SMBS, SMTP, SMTPS, TELNET and TFTP). The command is designed to
work without user interaction.
curl offers a busload of useful tricks like proxy support, user authenโ
tication, FTP upload, HTTP post, SSL connections, cookies, file transโ
fer resume, Metalink, and more. As you will see below, the number of
features will make your head spin!
โฆ with 3986 more linesๅฆๆๅๅงๆฐๆฎ้้ๅธธๅคง๏ผๆ่ ๅฎๆฏ่ฎธๅคๆไปถ็้ๅ๏ผๅ่ฏฅๆไปถๅฏ่ฝๆฏๅ็ผฉๆไปถใๅ ๅซ่ฎธๅค้ๅคๅผ็ๆฐๆฎ้๏ผๅฆๆๆฌๆไปถไธญ็ๅ่ฏๆ JSON ๆไปถไธญ็้ฎ๏ผ็นๅซ้ๅๅ็ผฉใ
ๅ็ผฉๆไปถๅธธ่ง็ๆไปถๆฉๅฑๅๆ๏ผ.tar.gzใ.zipๅ.rarใ่ฆ่งฃๅ็ผฉ่ฟไบๆไปถ๏ผไฝ ๅฏไปฅไฝฟ็จๅฝไปค่กๅทฅๅ
ทtarใunzipๅunrarใ๏ผๅฝ็ถ่ฟๆไธไบไธๅคชๅธธ่ง็ๆไปถๆฉๅฑๅ๏ผ่ฟไบๅฏ่ฝ้่ฆๅ
ถไปๅทฅๅ
ทๆฅๅค็ใ๏ผ
่ฎฉๆไปฌไปฅtar.gz๏ผ่ฏปไฝโgzipped tarballโ๏ผไธบไพใไธบไบๆๅๅไธบlogs.tar.gz็ๆกฃๆก๏ผไฝ ๅฐไฝฟ็จไปฅไธๅฝไปค่ก๏ผ
$ tar -xzf logs.tar.gz # โ โ ๅ็ผฉ็ๆถๅๅฐxzf่ฟไธไธช็ญ้้กน็ปๅๅจไธ่ตทๆฏๅพๅธธ่ง็๏ผๅฐฑๅๆๅจ่ฟ้ๅ็ไธๆ ทใๅฝ็ถไฝ ไนๅฏไปฅๅฐๅฎไปฌๅๅซๆๅฎไธบ-x -z -f๏ผไบๅฎไธ๏ผ่ฎธๅคๅฝไปคๅทฅๅ
ท้ฝๅ
่ฎธไฝ ็ฑๅไธชๅญ็ฌฆ็ปๅๆ้้กนใ
tarๅ ๅ
ถไผๅค็ๅฝไปค่กๅๆฐ่่ญๅๆญ่ใๅจ่ฟ็งๆ
ๅตไธ๏ผไธไธช้้กน-xใ-zๅ-f่กจๆtarๅฐไผ็จgzipไฝไธบ่งฃๅ็ผฉ็ฎๆณไปๅ็ผฉๆไปถไธญๆๅๆไปถใ
ไฝๆฏ๏ผ็ฑไบๆไปฌ่ฟไธ็ๆ่ฟไธชๅ็ผฉๆไปถ๏ผๆไปฅๆๅฅฝๅ
ๆฃๆฅไธไธๅฎ็ๅ
ๅฎน๏ผ่ฟๅฏไปฅ้่ฟ-t้้กน๏ผ่ไธๆฏ-x้้กน๏ผๆฅๅฎๆ๏ผ
$ tar -tzf logs.tar.gz | trim
E1FOSPSAYDNUZI.2020-09-01-00.0dd00628
E1FOSPSAYDNUZI.2020-09-01-00.b717c457
E1FOSPSAYDNUZI.2020-09-01-01.05f904a4
E1FOSPSAYDNUZI.2020-09-01-02.36588daf
E1FOSPSAYDNUZI.2020-09-01-02.6cea8b1d
E1FOSPSAYDNUZI.2020-09-01-02.be4bc86d
E1FOSPSAYDNUZI.2020-09-01-03.16f3fa32
E1FOSPSAYDNUZI.2020-09-01-03.1c0a370f
E1FOSPSAYDNUZI.2020-09-01-03.76df64bf
E1FOSPSAYDNUZI.2020-09-01-04.0a1ade1b
โฆ with 2427 more lines่ฟไธชๅ็ผฉๆไปถไธญๅ
ๅซไบๅพๅคๆไปถ๏ผๅนถไธๅฎไปฌไธๅจไธไธช็ฎๅฝไธญ๏ผๅ ๆญคไธบไบไฟๆๅฝๅ็ฎๅฝ็ๆดๆด๏ผๆๅฅฝ้ฆๅ
ไฝฟ็จmkdirๅๅปบไธไธชๆฐ็ฎๅฝ๏ผ็ถๅไฝฟ็จ-C้้กนๆๅๅ
ถไธญ็ๆไปถใ
$ mkdir logs
$ tar -xzf logs.tar.gz -C logs่ฎฉๆไปฌๆฅ็ไธไธๆไปถ็ๆฐ้ๅๅ ถ้จๅๅ ๅฎน๏ผ
$ ls logs | wc -l
2437
$ cat logs/* | trim
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stemโฆ
2020-09-01 00:51:54 SEA19-C1 391 206.55.174.150 GET โฆ
2020-09-01 00:54:59 CPH50-C2 384 82.211.213.95 GET โฆ
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stemโฆ
2020-09-01 00:04:28 DFW50-C1 391 2a03:2880:11ff:9::face:โฆ
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stemโฆ
2020-09-01 01:04:14 ATL56-C4 385 2600:1700:2760:da20:548โฆ
โฆ with 10279 more lines้ๅธธๅฅฝใ็ฐๅจ๏ผๆ็ฅ้ไฝ ๆณๆธ ็ๅ็ ็ฉถ่ฟไบๆฅๅฟๆไปถ๏ผไฝ้ฃๆฏไปฅๅๅจ็ฌฌไบ็ซ ๅ็ฌฌไธ็ซ ไธญ่ฎจ่ฎบ็ไบๆ ใ
้็ๆถ้ด็ๆจ็งป๏ผไฝ ไผไน ๆฏ่ฟไบ้้กน๏ผไฝๆๆณ็ปไฝ ็ไธไธชๆฏ่พๆนไพฟ็ๆฟไปฃ่ๆฌ๏ผๅฎไธ้่ฆ่ฎฐไฝไธๅ็ๅฝไปค่กๅทฅๅ
ทๅๅฎไปฌ็้้กน๏ผ่ฟไธชๆนไพฟ็่ๆฌๅซๅunpack๏ผๅฎๅฏไปฅ่งฃๅ็ผฉ่ฎธๅคไธๅ็ๆ ผๅผใunpackๆฅ็ไฝ ๆณ่ฆ่งฃๅ็ผฉ็ๆไปถ็ๆฉๅฑๅ๏ผๅนถ่ฐ็จ้ๅฝ็ๅฝไปค่กๅทฅๅ
ทใ็ฐๅจ๏ผไธบไบ่งฃๅ็ผฉ่ฟไธชๆไปถ๏ผไฝ ๅฏไปฅ่ฟ่ก๏ผ
$ unpack logs.tar.gzๅฏนไบ่ฎธๅคไบบๆฅ่ฏด๏ผMicrosoft Excel ๆไพไบไธ็ง็ด่ง็ๆนๅผๆฅๅค็ๅฐๅๆฐๆฎ้ๅนถๅฏนๅ
ถๆง่ก่ฎก็ฎใๅ ๆญค๏ผๅคง้ๆฐๆฎ่ขซๅตๅ
ฅๅฐ Microsoft Excel ็ตๅญ่กจๆ ผไธญใๆ นๆฎๆไปถๅ็ๆฉๅฑๅ๏ผ่ฟไบ็ตๅญ่กจๆ ผ่ฆไนไปฅไธๆ็ไบ่ฟๅถๆ ผๅผ๏ผ.xls๏ผๅญๅจ๏ผ่ฆไนไปฅๅ็ผฉ็ XML ๆไปถ็้ๅ๏ผ.xlsx๏ผๅญๅจใไฝ่ฟไธค็งๆ
ๅตไธ้ฝไธๅฉไบๅคงๅคๆฐๅฝไปค่กๅทฅๅ
ทไฝฟ็จ่ฟไบๆฐๆฎใๅฆๆไป
ไป
ๅ ไธบ่ฟไบๆไปทๅผ็ๆฐๆฎ้ไปฅ่ฟ็งๆนๅผๅญๅจ๏ผๆไปฌๅฐฑไธ่ฝไฝฟ็จๅฎไปฌไบ๏ผ้ฃ่ฟๅฐๆฏไธ็ง่ป่พฑใ
็นๅซๆฏๅฝไฝ ๅๅผๅงไฝฟ็จๅฝไปค่กๆถ๏ผไฝ ๅฏ่ฝไผๅฐ่ฏ้่ฟๅจ Microsoft Excel ๆๅผๆบ็ๆฌ๏ผๅฆ LibreOffice Calc๏ผไธญๆๅผ็ตๅญ่กจๆ ผ๏ผ็ถๅๆๅจๅฐๅ ถๅฏผๅบไธบ CSV ๆ ผๅผ๏ผไป่ๅฐ็ตๅญ่กจๆ ผ่ฝฌๆขไธบ CSV ๆ ผๅผใ่ฝ็ถ่ฟไนๆฏไธไธช่งฃๅณๆนๆก๏ผไฝ็ผบ็นๆฏๅฎไธ่ฝๅพๅฅฝๅฐๆฉๅฑๅฐๅคไธชๆไปถ๏ผๅนถไธไธ่ฝ่ชๅจๅใๆญคๅค๏ผๅฝไฝ ๅจๆๅกๅจไธๅทฅไฝๆถ๏ผๅพๅฏ่ฝๆฒกๆ่ฟๆ ท็ๅบ็จๅฏ็จใ็ธไฟกๆ๏ผๅฝไปค่กๆฏไธไธชๆดๅฅฝ็่งฃๅณๆนๆกใ
ๅนธ่ฟ็ๆฏ๏ผๆไปฌๆไธไธชๅไธบin2csv็ๅฝไปค่กๅทฅๅ
ท๏ผๅฎๅฏไปฅๅฐ Microsoft Excel ็ตๅญ่กจๆ ผ่ฝฌๆขๆ CSV ๆไปถใCSV ๆ็ๆฏ้ๅทๅ้็ๆฐๅผ๏ผไฝฟ็จ CSV ๆไปถๅฏ่ฝไผๅพๆฃๆ๏ผๅ ไธบๅฎ็ผบไนๆญฃๅผ็่ง่ใYakov Shafranovich ๅฐ CSV ๆ ผๅผๅฎไนไธบไปฅไธไธ็น๏ผ
- ๆฏๆก่ฎฐๅฝไฝไบๅ็ฌ็ไธ่ก๏ผ็ฑๆข่ก็ฌฆ๏ผ
LF๏ผๅ้ใไพๅฆ๏ผไธ้ข็ CSV ๆไปถๅ ๅซไบๅ ณไบๅฟ่ ็ฅ้พ็้่ฆไฟกๆฏ๏ผ
$ bat -A tmnt-basic.csv # โ
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: tmnt-basic.csv
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ Leonardo,Leo,blue,twoยทninjakensโ
2 โ Raphael,Raph,red,pairยทofยทsaiโ
3 โ Michelangelo,MikeyยทorยทMike,orange,pairยทofยทnunchakuโ
4 โ Donatello,DonnieยทorยทDon,purple,staffโ
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ -A้้กนไฝฟbatๆพ็คบๆๆไธๅฏๆๅฐ็ๅญ็ฌฆ๏ผๅฆ็ฉบๆ ผใๅถ่กจ็ฌฆๅๆข่ก็ฌฆใ
- ๆไปถไธญ็ๆๅไธๆก่ฎฐๅฝๅฏ่ฝๆไนๅฏ่ฝๆฒกๆ็ปๆๆข่ก็ฌฆ๏ผไพๅฆ๏ผ
$ bat -A tmnt-missing-newline.csv
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: tmnt-missing-newline.csv
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ Leonardo,Leo,blue,twoยทninjakensโ
2 โ Raphael,Raph,red,pairยทofยทsaiโ
3 โ Michelangelo,MikeyยทorยทMike,orange,pairยทofยทnunchakuโ
4 โ Donatello,DonnieยทorยทDon,purple,staff
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ- ๆไปถ็็ฌฌไธ่กๅฏ่ฝไผๅบ็ฐไธไธชๆ ้ข่ก๏ผๅ ถๆ ผๅผไธๆฎ้่ฎฐๅฝ่ก็ธๅใ่ฏฅๆ ้ขๅฐๅ ๅซไธๆไปถไธญ็ๅญๆฎต็ธๅฏนๅบ็ๅ็งฐ๏ผๅนถไธๅบ่ฏฅๅ ๅซไธๆไปถๅ ถไฝ้จๅไธญ็่ฎฐๅฝ็ธๅๆฐ้็ๅญๆฎตใไพๅฆ๏ผ
$ bat -A tmnt-with-header.csv
โโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ File: tmnt-with-header.csv
โโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
1 โ name,nickname,mask_color,weaponโ
2 โ Leonardo,Leo,blue,twoยทninjakensโ
3 โ Raphael,Raph,red,pairยทofยทsaiโ
4 โ Michelangelo,MikeyยทorยทMike,orange,pairยทofยทnunchakuโ
5 โ Donatello,DonnieยทorยทDon,purple,staffโ
โโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโๅฆไฝ ๆ่ง๏ผ้ป่ฎคๆ
ๅตไธ๏ผCSV ไธๅคชๅฏ่ฏปใไฝ ๅฏไปฅ้่ฟ็ฎก้ๅฐๆฐๆฎไผ ่พๅฐไธไธชๅไธบcsvlook็ๅทฅๅ
ท๏ผๅฎไผๅพๅฅฝๅฐๅฐๆฐๆฎๆ ผๅผๅๆ่กจๆ ผใๅฆๆ CSV ๆฐๆฎๆฒกๆๅคด๏ผๆฏๅฆtmnt-missing-newline.csv๏ผ้ฃไนไฝ ้่ฆๆทปๅ -H้้กน๏ผๅฆๅ็ฌฌไธ่กๅฐ่ขซ่งฃ้ไธบๅคดใ
$ csvlook tmnt-with-header.csv
โ name โ nickname โ mask_color โ weapon โ
โโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโค
โ Leonardo โ Leo โ blue โ two ninjakens โ
โ Raphael โ Raph โ red โ pair of sai โ
โ Michelangelo โ Mikey or Mike โ orange โ pair of nunchaku โ
โ Donatello โ Donnie or Don โ purple โ staff โ
$ csvlook tmnt-basic.csv
โ Leonardo โ Leo โ blue โ two ninjakens โ
โโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโผโโโโโโโโโผโโโโโโโโโโโโโโโโโโโค
โ Raphael โ Raph โ red โ pair of sai โ
โ Michelangelo โ Mikey or Mike โ orange โ pair of nunchaku โ
โ Donatello โ Donnie or Don โ purple โ staff โ
$ csvlook -H tmnt-missing-newline.csv # โ
โ a โ b โ c โ d โ
โโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโผโโโโโโโโโผโโโโโโโโโโโโโโโโโโโค
โ Leonardo โ Leo โ blue โ two ninjakens โ
โ Raphael โ Raph โ red โ pair of sai โ
โ Michelangelo โ Mikey or Mike โ orange โ pair of nunchaku โ
โ Donatello โ Donnie or Don โ purple โ staff โโ -H้้กน่กจ็คบ CSV ๆไปถไธญๆฒกๆๆ ้ข่กใ
่ฎฉๆไปฌ็จไธไธช็ตๅญ่กจๆ ผๆฅๆผ็คบไธไธin2csv๏ผ่ฟไธช่กจๆ ผๅ
ๅซไบไธๅนดไธๅบฆ็่ทๅ
ฐ้ฉฌๆๆพๅนฟๆญ่็ฎ 2000 ๅผบไธญ 2000 ้ฆๆๆต่ก็ๆญๆฒใ่ฆๆๅๅฎ็ๆฐๆฎ๏ผไฝ ๅฏไปฅๅฆไธ่ฐ็จin2csv:
$ in2csv top2000.xlsx | tee top2000.csv | trim
NR.,ARTIEST,TITEL,JAAR
1,Danny Vera,Roller Coaster,2019
2,Queen,Bohemian Rhapsody,1975
3,Eagles,Hotel California,1977
4,Billy Joel,Piano Man,1974
5,Led Zeppelin,Stairway To Heaven,1971
6,Pearl Jam,Black,1992
7,Boudewijn de Groot,Avond,1997
8,Coldplay,Fix You,2005
9,Pink Floyd,Wish You Were Here,1975
โฆ with 1991 more linesDanny Vera ๆฏ่ฐ๏ผๅฝ็ถ๏ผๆๅๆฌข่ฟ็ๆญๆฒๅบ่ฏฅๆฏๆณข่ฅฟ็ฑณไบ็ๆณๆฒใๅฏ๏ผ่ณๅฐ Queen ๅจๅ 2000 ๅไธญๅบ็ฐไบๅพๅคๆฌก๏ผๆไปฅๆไนไธๅบ่ฏฅๆฑๆจ๏ผ
$ csvgrep top2000.csv --columns ARTIEST --regex '^Queen$' | csvlook -I # โ
โ NR. โ ARTIEST โ TITEL โ JAAR โ
โโโโโโโโผโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโค
โ 2 โ Queen โ Bohemian Rhapsody โ 1975 โ
โ 11 โ Queen โ Love Of My Life โ 1975 โ
โ 46 โ Queen โ Innuendo โ 1991 โ
โ 55 โ Queen โ Don't Stop Me Now โ 1979 โ
โ 70 โ Queen โ Somebody To Love โ 1976 โ
โ 85 โ Queen โ Who Wants To Live Forever โ 1986 โ
โ 89 โ Queen โ The Show Must Go On โ 1991 โ
โ 131 โ Queen โ Killer Queen โ 1974 โ
โฆ with 24 more linesโ --regex้้กนๅ็ๅผๆฏไธไธชๆญฃๅ่กจ่พพๅผ๏ผๆ regex๏ผใ่ฟๆฏไธ็งๅฎไนๆจกๅผ็็นๆฎ่ฏญๆณใๅ ไธบๆๅชๆณๅน้
ไธQueenๅฎๅ
จๅน้
็่บๆฏๅฎถ๏ผๆไปฅๆไฝฟ็จๆๅ
ฅ็ฌฆๅท๏ผ^๏ผๅ็พๅ
็ฌฆๅท๏ผ$๏ผๆฅๅน้
ๅARTIESTไธญๅผ็ๅผๅงๅ็ปๆใ
้กบไพฟ่ฏดไธไธ๏ผๅทฅๅ
ทin2csvใcsvgrepๅcsvlook้ฝๆฏ CSVkit ็ไธ้จๅ๏ผCSVkit ๆฏๅค็ CSV ๆฐๆฎ็ๅฝไปค่กๅทฅๅ
ท็้ๅใ
ๆไปถ็ๆ ผๅผๆฏ็ฑๆฉๅฑๅ่ชๅจๅณๅฎ็๏ผๆฌไพไธญๆฏ.xlsxใๅฆๆไฝ ่ฆๅฐๆฐๆฎๅฏผๅ
ฅin2csv๏ผไฝ ๅฟ
้กปๆ็กฎๆๅฎๆ ผๅผใ
ไธไธช็ตๅญ่กจๆ ผๅฏไปฅๅ
ๅซๅคไธชๅทฅไฝ่กจใ้ป่ฎคๆ
ๅตไธ๏ผin2csvๆๅ็ฌฌไธไธชๅทฅไฝ่กจใๅฆๆ่ฆๆๅไธๅ็ๅทฅไฝ่กจ๏ผ้ฃไน้่ฆๅฐๅทฅไฝ่กจ็ๅ็งฐไผ ้็ป--sheet้้กนใๅฆๆไฝ ไธ็กฎๅฎๅทฅไฝ่กจๅซไปไน๏ผไฝ ๅฏไปฅไฝฟ็จ--names้้กนๆฅ็๏ผๅฎไผๆๅฐๅบๆๆๅทฅไฝ่กจ็ๅ็งฐใ่ฟ้ๆไปฌ็ๅฐtop2000.xlsxๅชๆไธๅผ ่กจ๏ผๅไธบBlad1๏ผ่ทๅ
ฐ่ฏญ๏ผๆๆๆฏSheet1๏ผใ
$ in2csv --names top2000.xlsx
Blad1่ฎธๅคๅ ฌๅธๅฐไปไปฌ็ๆฐๆฎๅญๅจๅจๅ ณ็ณปๆฐๆฎๅบไธญใๅฐฑๅ็ตๅญ่กจๆ ผไธๆ ท๏ผๅฆๆๆไปฌๅฏไปฅไปๅฝไปค่ก่ทๅพ่ฟไบๆฐๆฎ๏ผ้ฃๅฐฑๅคชๅฅฝไบใ
ๅ ณ็ณปๆฐๆฎๅบ็ไพๅญๆ MySQLใPostgreSQL ๅ SQLiteใ่ฟไบๆฐๆฎๅบ็ๆฅๅฃๆนๅผ็จๅพฎๆไบไธๅใๆไบๆไพๅฝไปค่กๅทฅๅ ทๆๅฝไปค่ก็้ข๏ผๆไบๅไธๆไพใๆญคๅค๏ผๅฝๆถๅๅฐๅฎไปฌ็ไฝฟ็จๅ่พๅบๆถ๏ผๆ ผๅผไธๆฏๅพไธ่ดใ
ๅนธ่ฟ็ๆฏ๏ผๆไธไธชๅไธบsql2csv็ๅฝไปค่กๅทฅๅ
ทไธ้จ็จๆฅๅ่ฟไธชไบ๏ผๅฎไนๆฏ CSVkit ็ไธ้จๅใๅฎ้่ฟไธไธชๅ
ฌๅ
ฑๆฅๅฃไธ่ฎธๅคไธๅ็ๆฐๆฎๅบๅๅๅทฅไฝ๏ผๅ
ๆฌ MySQLใOracleใPostgreSQLใSQLiteใMicrosoft SQL Server ๅ Sybaseใsql2csv็่พๅบ๏ผ้กพๅๆไน๏ผๅฐฑๆฏ CSV ๆ ผๅผ็ใ
ๆไปฌๅฏไปฅ้่ฟๅฏนๅ
ณ็ณปๆฐๆฎๅบๆง่กSELECTๆฅ่ฏขๆฅ่ทๅๆฐๆฎใ๏ผsql2csvไนๆฏๆINSERTใUPDATEๅDELETEๆฅ่ฏข๏ผไฝ่ฟไธๆฏๆฌ็ซ ็็ฎ็ใ๏ผ
sql2csv้่ฆไธคไธชๅๆฐ๏ผ--db๏ผๆๅฎๆฐๆฎๅบ URL๏ผๅฝขๅผไธ่ฌๆฏ๏ผdialect+driver://username:password@host:port/database๏ผ--query๏ผๅ
ๅซSELECTๆฅ่ฏขใไพๅฆ๏ผๆๅฎไธไธชๅ
ๅซๆฅ่ช R ็ๆ ๅๆฐๆฎ้็ SQLite ๆฐๆฎๅบ๏ผๆๅฏไปฅไป่กจmtcarsไธญ้ๆฉๆๆ่ก๏ผๅนถๆmpgๅๅฏนๅฎไปฌ่ฟ่กๆๅบ๏ผๅฆไธๆ็คบ๏ผ
$ sql2csv --db 'sqlite:///r-datasets.db' \
> --query 'SELECT row_names AS car, mpg FROM mtcars ORDER BY mpg' | csvlook
โ car โ mpg โ
โโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโค
โ Cadillac Fleetwood โ 10.4 โ
โ Lincoln Continental โ 10.4 โ
โ Camaro Z28 โ 13.3 โ
โ Duster 360 โ 14.3 โ
โ Chrysler Imperial โ 14.7 โ
โ Maserati Bora โ 15.0 โ
โ Merc 450SLC โ 15.2 โ
โ AMC Javelin โ 15.2 โ
โฆ with 24 more lines่ฟไธชไพๅญไธญ SQLite ๆฐๆฎๅบๆฏไธไธชๆฌๅฐๆไปถ๏ผๆไปฅๅจ่ฟ้ๆไธ้่ฆๆๅฎไปปไฝ็จๆทๅใๅฏ็ ๆไธปๆบใๅฆๆไฝ ๆณๆฅ่ฏขไฝ ้ไธป็ๆฐๆฎๅบ๏ผไฝ ๅฝ็ถ้่ฆ็ฅ้ๅฆไฝ่ฎฟ้ฎๅฎ๏ผๅนถไธไฝ ้่ฆๅพๅฐๆ้ใ
ๅจไธไธ่ไธญ๏ผๆ่งฃ้ไบๅฆไฝไปไบ่็ฝไธไธ่ฝฝๆไปถใไปไบ่็ฝไธๆฟๆฐๆฎ็ๅฆไธ็งๆนๅผๆฏ้่ฟ Web API๏ผๅฎไปฃ่กจๅบ็จ็ผ็จๆฅๅฃ๏ผAPI ๆฐ้ๆญฃๅจไปฅ่ถๆฅ่ถๅฟซ็้ๅบฆๅข้ฟ๏ผ่ฟๅฏนๆไปฌๆฐๆฎ็งๅญฆๅฎถๆฅ่ฏดๆๅณ็ๅคง้ๆ่ถฃ็ๆฐๆฎใ
Web API ๅนถไธๆๅณ็่ฆไปฅๆผไบฎ็ๅธๅฑๅ็ฐ๏ผๆฏๅฆ็ฝ็ซใ็ธๅ๏ผๅคงๅคๆฐ Web API ไปฅ็ปๆๅๆ ผๅผ่ฟๅๆฐๆฎ๏ผๆฏๅฆ JSON ๆ XMLใไปฅ็ปๆๅ็ๅฝขๅผไฟๅญๆฐๆฎ็ๅฅฝๅคๆฏๆฐๆฎๅฏไปฅๅพๅฎนๆๅฐ่ขซๅ
ถไปๅทฅๅ
ทๅค็๏ผๆฏๅฆjqใไพๅฆ๏ผไพๅญไธญ็ API ๅ
ๅซๅคง้ๅ
ณไบ George R.R. Martin ่ๆ็ไธ็็ไฟกๆฏ๏ผ่ใๆๅ็ๆธธๆใไธไนฆๅ็ต่งๅงๅฐฑๅ็ๅจ่ฟไธช่ๆไธ็ไธญ๏ผๅฎไปฅไธ้ข็ JSON ็ปๆ่ฟๅๆฐๆฎ๏ผ
$ curl -s "https://anapioficeandfire.com/api/characters/583" | jq '.'
{
"url": "https://anapioficeandfire.com/api/characters/583",
"name": "Jon Snow",
"gender": "Male",
"culture": "Northmen",
"born": "In 283 AC",
"died": "", # โ
"titles": [
"Lord Commander of the Night's Watch"
],
"aliases": [
"Lord Snow",
"Ned Stark's Bastard",
"The Snow of Winterfell",
"The Crow-Come-Over",
"The 998th Lord Commander of the Night's Watch",
"The Bastard of Winterfell",
"The Black Bastard of the Wall",
"Lord Crow"
],
"father": "",
"mother": "",
"spouse": "",
"allegiances": [
"https://anapioficeandfire.com/api/houses/362"
],
"books": [
"https://anapioficeandfire.com/api/books/5"
],
"povBooks": [
"https://anapioficeandfire.com/api/books/1",
"https://anapioficeandfire.com/api/books/2",
"https://anapioficeandfire.com/api/books/3",
"https://anapioficeandfire.com/api/books/8"
],
"tvSeries": [
"Season 1",
"Season 2",
"Season 3",
"Season 4",
"Season 5",
"Season 6"
],
"playedBy": [
"Kit Harington"
]
}โ ๅง้๏ผ่ฟไธชๆฐๆฎๅนถไธๅฎๅ จๆฏๆๆฐ็ใ
ๆฐๆฎ้่ฟ็ฎก้ไผ ่พๅฐๅฝไปค่กๅทฅๅ
ทjq๏ผ่ฟๅชๆฏไธบไบไปฅไธ็งๆผไบฎ็ๆนๅผๆพ็คบๅฎใjqๆๆดๅคๆธ
็ๅๆข็ดข็ๅฏ่ฝๆง๏ผๆไปฌๅฐๅจ็ฌฌไบ็ซ ๅ็ฌฌไธ็ซ ไธญๆข็ดขใ
ไธไบ Web API ่ฆๆฑไฝ ๅจ่ฏทๆฑๅฎไปฌ็่พๅบไนๅ่ฟ่ก่บซไปฝ้ช่ฏ๏ผๅณ่ฏๆไฝ ็่บซไปฝ๏ผใๆๅ ็งๆนๆณๅฏไปฅๅๅฐ่ฟไธ็นใไธไบ Web API ไฝฟ็จ API ๅฏๅ๏ผ่ๅฆไธไบไฝฟ็จ OAuth ๅ่ฎฎใๅจ่ฟ้๏ผNews API๏ผไธไธช็ฌ็ซ็ๆ ้ขๅๆฐ้ปๆ็ซ ๆฅๆบ๏ผๅฐฑๆฏไธไธชๅพๅฅฝ็ไพๅญใ่ฎฉๆไปฌ็็ๅฝไฝ ่ฏๅพๅจๆฒกๆ API ้ฎ็ๆ ๅตไธ่ฎฟ้ฎ่ฟไธช API ๆถไผๅ็ไปไน๏ผ
$ curl -s "http://newsapi.org/v2/everything?q=linux" | jq .
{
"status": "error", "code": "apiKeyMissing",
"message": "Your API key is missing. Append this to the URL with the apiKey pa
ram, or use the x-api-key HTTP header."
}ๅฏ๏ผ้ฃๆฏๆๆไนไธญ็ใ้กบไพฟ่ฏดไธไธ๏ผ้ฎๅทๅ้ข็้จๅๆฏๆไปฌไผ ้ๆฅ่ฏขๅๆฐ็ๅฐๆน๏ผ่ฟไนๆฏไฝ ้่ฆๆๅฎ API ๅฏๅ็ๅฐๆนใไฝๆฏๆๆณๅฏน่ชๅทฑ็ API ๅฏๅไฟๅฏ๏ผๆไปฅๆ้่ฟ่ฏปๅๆไปถ็ๆนๅผๅฐไฟกๆฏๆๅ ฅ่ฟๅปใ
$ curl -s "http://newsapi.org/v2/everything?q=linux&apiKey=$(< /data/.secret/new
sapi.org_apikey)" |
> jq '.' | trim 30
{
"status": "ok",
"totalResults": 9178,
"articles": [
{
"source": {
"id": "the-verge",
"name": "The Verge"
},
"author": "Sean Hollister",
"title": "Bungie rejects Steam Deck, threatens to ban Destiny 2 players tโฆ
"description": "Not only is Bungie apparently not making Destiny 2 compatโฆ
"url": "https://www.theverge.com/22957294/bungie-destiny-2-steam-deck-gamโฆ
"urlToImage": "https://cdn.vox-cdn.com/thumbor/PTFqHfu8ezNz4QrLf_ugVteHNVโฆ
"publishedAt": "2022-03-01T23:45:56Z",
"content": "Image: Bungie\r\n\n \n\n When will Bungie let Destiny 2 come โฆ
},
{
"source": {
"id": "ars-technica",
"name": "Ars Technica"
},
"author": "Scharon Harding",
"title": "System76 Linux workstation looks ready for gaming, too",
"description": "A 144 Hz screen, colorful keyboard, and Ethernet can appeโฆ
"url": "https://arstechnica.com/gadgets/2022/02/system76-linux-laptop-uniโฆ
"urlToImage": "https://cdn.arstechnica.net/wp-content/uploads/2022/02/lisโฆ
"publishedAt": "2022-02-03T16:49:47Z",
"content": "13 with 13 posters participating\r\nSystem76's latest Linux lโฆ
},
โฆ with 236 more linesไฝ ๅฏไปฅๅจ News API ็็ฝ็ซ่ทๅพ่ชๅทฑ็ API ๅฏๅใ
ไธไบ Web API ไปฅๆต็ๆนๅผ่ฟๅๆฐๆฎใ่ฟๆๅณ็ไธๆฆไฝ ่ฟๆฅๅฐๅฎ๏ผๆฐๆฎๅฐ็ปง็ปญๆถๅ ฅ๏ผ็ดๅฐ่ฟๆฅ่ขซๅ ณ้ญใไธไธชไผๆๅจ็ฅ็ไพๅญๆฏ Twitter โfire hoseโ๏ผๅฎไธๆญๅฐๅไธ็ๅๅฐๅ้ๆๆ็ๆจๆใๅนธ่ฟ็ๆฏ๏ผๅคงๅคๆฐๅฝไปค่กๅทฅๅ ทไนไปฅๆตๆนๅผ่ฟ่กใ
ไพๅฆ๏ผ่ฎฉๆไปฌๆฅ็ไธไธช 10 ็ง้็ Wikimedia ๆตๅชไฝ API ็คบไพ๏ผ
$ curl -s "https://stream.wikimedia.org/v2/stream/recentchange" |
> sample -s 10 > wikimedia-stream-sample่ฟไธช็นๅฎ็ API ่ฟๅๅฏน Wikipedia ๅ Wikimedia ็ๅ
ถไปๅฑๆงๆๅ็ๆๆๆดๆนใๅฝไปค่กๅทฅๅ
ทsample็จไบๅจ 10 ็งๅๅ
ณ้ญ่ฟๆฅ๏ผๆไปฌไนๅฏไปฅ้่ฟๆไธCtrl-Cๅ้ไธญๆญๆฅๆๅจๅ
ณ้ญ่ฟๆฅใ่พๅบ่ขซไฟๅญๅฐๆไปถwikimedia-stream-sample๏ผ่ฎฉๆไปฌ็จtrimๆฅไธ็ชฅ็ฉถ็ซ๏ผ
$ < wikimedia-stream-sample trim
:ok
event: message
id: [{"topic":"eqiad.mediawiki.recentchange","partition":0,"timestamp":16101133โฆ
data: {"$schema":"/mediawiki/recentchange/1.0.0","meta":{"uri":"https://en.wikiโฆ
event: message
id: [{"topic":"eqiad.mediawiki.recentchange","partition":0,"timestamp":16101133โฆ
data: {"$schema":"/mediawiki/recentchange/1.0.0","meta":{"uri":"https://www.wikโฆ
โฆ with 1078 more lines้่ฟsedๅjq๏ผๆไปฌๅฐฑๅฏไปฅๆธ
็่ฟไบๆฐๆฎ๏ผไป่็ๅฐ่ฑๆ็็ปดๅบ็พ็งๅ็็ๅๅ๏ผ
$ < wikimedia-stream-sample sed -n 's/^data: //p' | # โ
> jq 'select(.type == "edit" and .server_name == "en.wikipedia.org") | .title' # โ
"Odion Ighalo"
"Hold Up (song)"
"Talk:Royal Bermuda Yacht Club"
"Jenna Ushkowitz"
"List of films released by Yash Raj Films"
"SP.A"
"Odette (musician)"
"Talk:Pierre Avoi"
"User:Curlymanjaro/sandbox3"
"List of countries by electrification rate"
"Grieg (crater)"
"Gorman, Edmonton"
"Khabza"
"QAnon"
"Khaw Boon Wan"
"Draft:Oggy and the Cockroaches (1975 TV series)"
"Renzo Reggiardo"
"Greer, Arizona"
"National Curriculum for England"
"Mod DB"
"Jordanian Pro League"
"List of foreign Serie A players"โ ่ฟไธชsed่กจ่พพๅผ่กจ็คบๅชๆๅฐไปฅdata:ๅผๅคด็่ก๏ผๆๅฐๅๅทๅ็้จๅ๏ผๆฐๅฅฝๆฏ JSONใ
โ ่ฟไธชjq่กจ่พพๅผๆๅฐๅ
ทๆๆไธชtypeๅserver_name็ JSON ๅฏน่ฑก็title้ฎใ
่ฏดๅฐๆตๅชไฝ๏ผไฝ ็ฅ้ไฝ ๅฏไปฅไฝฟ็จtelnetๅ
่ดนๆญๆพใๆ็ๅคงๆ๏ผ็ฌฌๅ้โโๆฐ็ๅธๆใๅ๏ผ
$ telnet towel.blinkenlights.nl่ฟไบไธไผๅฟ๏ผๆไปฌ็ๅฐ Han Solo ๅ ๅผๆชไบ๏ผ
-=== `"',
I'll bet you ""o o O O|)
have! _\ -/_ _\o/ _
|| || |* /|\ / |\
\\ || *** //| | |\\
\\o=*********** // | | | ||
|\(#'***\\ -==# | | | ||
|====|* ') '\ |====| /#
|/|| | | || | "
( )( ) | || |
|-||-| | || |
| || | | || |
________________[_][__\________________/__)(_)_____________________
ๅฝ็ถ๏ผ่ฟๅฏ่ฝไธๆฏไธไธชๅฅฝ็ๆฐๆฎๆฅๆบ๏ผไฝๅจ่ฎญ็ปไฝ ็ๆบๅจๅญฆไน ๆจกๅ็ๅๆถๆฌฃ่ตไธ้จ่็ปๅ ธไนๆฒกไปไน้ใ
ๆญๅไฝ ๏ผไฝ ๅทฒ็ปๅฎๆไบ OSEMN ๆจกๅ็็ฌฌไธๆญฅใไฝ ๅทฒ็ปๅญฆไน ไบๅ็ง่ทๅๆฐๆฎ็ๆนๆณ๏ผไปไธ่ฝฝๅฐๆฅ่ฏขๅ ณ็ณปๆฐๆฎๅบใๅจไธไธ็ซ ๏ผไนๆฏไธญ้ด็ซ ่๏ผๆๅฐๆไฝ ๅฆไฝๅๅปบไฝ ่ชๅทฑ็ๅฝไปค่กๅทฅๅ ทใๅฆๆไฝ ่ฟซไธๅๅพ ๅฐๆณไบ่งฃๆฐๆฎๆธ ็๏ผๅฏไปฅ่ทณ่ฟ่ฟไธๆญฅ๏ผ็ดๆฅ่ฟๅ ฅ็ฌฌไบ็ซ ๏ผOSEMN ๆจกๅ็็ฌฌไบๆญฅ๏ผใ
- ๅฏปๆพไธไธชๆฐๆฎ้ๆฅ็ปไน ๏ผGitHub ็ฅ่ฏๅบ Awesome Public Datasets ๅๅบไบๆฐ็พไธชๅ ฌๅผๅฏ็จ็้ซ่ดจ้ๆฐๆฎ้
- ๆ่ ไฝ ๆดๆฟๆ็จ API ๆฅ็ปไน ๏ผGitHub ๅบ Public API ๅๅบไบๅพๅคๅ ่ดน APIใCity Bikes ๅ The One API ๆฏๆ็ๆ็ฑ
- ็ผๅ SQL ๆฅ่ฏขไปๅ ณ็ณปๆฐๆฎๅบไธญ่ทๅๆฐๆฎๆฏไธ้กน้่ฆ็ๆ่ฝใBen Forta ็ใSQL in 10 Minutes a Dayใไธไนฆ็ๅ 15 ่ฏพ่ฎฒไบ SELECT ่ฏญๅฅๅๅ ถ่ฟๆปคใๅ็ปๅๆๅบๅ่ฝ