skete-haskell serves empty index file
Symptoms: cabal update
succeeds, but the index file it receives (1k total) is empty.
Don't know if it matters, but the corporate security scanner bombards this skete
server , like (taken from the nohup.log
):
GET /
Accept:
Status: 404 Not Found 0.000016892s
GET /
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000012796s
GET /login.php
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000011494s
GET /this_server/all_settings.shtml
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000012897s
GET /image/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000006689s
GET /imagenes/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000006033s
GET /imagery/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.00000566s
. . . . .
GET /import/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000005319s
GET /impreso/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000005504s
GET /inc/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000005589s
GET /info/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000005697s
GET /information/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000010759s
. . . . .
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000005447s
GET /internal/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000005641s
GET /inventory/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000005365s
GET /invitado/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000005524s
GET /isapi/
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
Status: 404 Not Found 0.000009987s
. . . . .
[and a ton more]
Being able to turn TLS on, and enforce client authentication would've blocked this kind of abuse, but that's a different story.
Here's what's happening:
$ cabal --verbose --config-file ~/.cabal/skete-config update
Project settings changed, reconfiguring...
Downloading the latest package list from ll-skete
/opt/local/bin/curl 'http://g53blumenthal.llan.ll.mit.edu:12344/packages/archive/00-index.tar.gz' --output /Users/ur20980/.cabal/packages/ll-skete/00-index.tar58920-0.gz --location --write-out '%{http_code}' --user-agent 'cabal-install/3.2.0.0 (osx; x86_64)' --silent --show-error --dump-header /Users/ur20980/.cabal/packages/ll-skete/curl-headers58920-1.txt
Downloaded to /Users/ur20980/.cabal/packages/ll-skete/00-index.tar.gz
Updating index cache file
/Users/ur20980/.cabal/packages/ll-skete/00-index.cache ...
Index cache updated to index-state
$ ll /Users/ur20980/.cabal/packages/ll-skete/00-index.tar.gz
-rw------- 1 ur20980 179427977 1024 Jun 28 21:20 /Users/ur20980/.cabal/packages/ll-skete/00-index.tar.gz
$ tar tzvf /Users/ur20980/.cabal/packages/ll-skete/00-index.tar.gz
$
Here's what the log shows on the server that runs skete
:
$ more nohup.out
GET /packages/archive/00-index.tar.gz
Accept: */*
Status: 200 OK 0.000393717s
GET /packages/archive/00-index.tar.gz
Accept: */*
Status: 200 OK 0.000073136s
Here's how I start it:
$ alias start-skete
alias start-skete='cd $HOME/src/skete-haskell && nohup bin/skete-haskell-exe serve --port 12344 --repo ./haskell-packages/ &'
$
Here's the script I use to update the local repo from Hackage (I haven't tried to wipe the existing packages, and start from scratch, git init
etc.):
#!/bin/bash -e
if [ -z "${SKETE_HASKELL_DIR}" ]; then
export SKETE_HASKELL_DIR="${HOME}/src/skete-haskell"
fi
pushd ${SKETE_HASKELL_DIR}
# Optional: Create a location to save tarballs downloaded from hackage to avoid redownloading on future imports.
mkdir -p hackage-cache
# Prepare the skete-haskell git repository
mkdir -p haskell-packages
# Initialize git of Hackage mirror, if not already initialized
if [ ! -r haskell-packages/.git/config ]; then
pushd haskell-packages
git init
popd
fi
# fetch the current index tarball from hackage to tell skete what to import
wget https://hackage.haskell.org/01-index.tar
# import the current hackage state (This will take about an hour, depending on internet and disk speed)
bin/skete-haskell-exe import --cache hackage-cache --repo haskell-packages/.git
# Optional: pack the newly created repository so it takes less space on disk.
pushd haskell-packages
git gc --aggressive --prune=now
popd
rm 01-index.tar
It seems to work, based on this:
$ pwd
/home/ur20980/src/skete-haskell
$ ll
total 16776
drwxr-xr-x. 10 ur20980 4096 Jun 28 21:19 ./
drwxr-xr-x. 10 ur20980 4096 Jun 13 22:12 ../
drwxr-xr-x. 2 ur20980 31 Mar 25 05:44 bin/
-rw-r--r--. 1 ur20980 1787 Mar 26 07:51 cabal.project
-rw-r--r--. 1 ur20980 0 Mar 25 05:43 cabal.project.local
-rw-r--r--. 1 ur20980 0 Mar 25 05:43 cabal.project.local~
drwxr-xr-x. 5 ur20980 43 Mar 15 11:43 dist-newstyle/
drwxr-xr-x. 2 ur20980 23 Mar 24 19:53 docs/
drwxr-xr-x. 8 ur20980 203 Jun 26 00:27 .git/
drwxr-xr-x. 2 ur20980 98304 Jun 26 21:46 hackage-cache/
drwxr-xr-x. 57 ur20980 4096 Mar 23 21:50 haskell-packages/
-rw-r--r--. 1 ur20980 35688 Mar 18 11:33 install-log2.txt
-rw-r--r--. 1 ur20980 1510 Mar 15 09:20 LICENSE
-rw-r--r--. 1 ur20980 3680 Mar 15 11:44 logo.png
-rw-------. 1 ur20980 236 Jun 28 21:20 nohup.out
-rw-------. 1 ur20980 847605 Mar 31 10:24 nohup.out1
-rw-------. 1 ur20980 841140 Apr 8 18:05 nohup.out2
-rw-------. 1 ur20980 15267475 Jun 26 17:22 nohup.out3
-rw-r--r--. 1 ur20980 46 Mar 15 09:20 Setup.hs
-rw-r--r--. 1 ur20980 2581 Mar 26 07:51 skete-haskell.cabal
drwxr-xr-x. 3 ur20980 26 Mar 15 11:44 src/
drwxr-xr-x. 2 ur20980 113 Mar 25 05:43 tools/
$ du -sh hackage-cache haskell-packages/
382M hackage-cache
2.6G haskell-packages/
$
For comparison, this is what I have for Hackage:
$ cabal --verbose update
Project settings changed, reconfiguring...
Downloading the latest package list from hackage.haskell.org
Selected mirror https://hackage.haskell.org/
Downloading timestamp
/opt/local/bin/curl 'https://hackage.haskell.org/timestamp.json' --output /var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/transportAdapterGet59618-1 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.2.0.0 (osx; x86_64)' --silent --show-error --dump-header /var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/curl-headers59618-2.txt
Downloading snapshot
/opt/local/bin/curl 'https://hackage.haskell.org/snapshot.json' --output /var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/transportAdapterGet59618-4 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.2.0.0 (osx; x86_64)' --silent --show-error --dump-header /var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/curl-headers59618-5.txt
Updating index
/opt/local/bin/curl 'https://hackage.haskell.org/01-index.tar.gz' --output /var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/transportAdapterGet59618-7 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.2.0.0 (osx; x86_64)' --silent --show-error --dump-header /var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/curl-headers59618-8.txt --header 'Range: bytes=90254600-90553985'
Waiting to acquire cache lock on
/Users/ur20980/.cabal/packages/hackage.haskell.org/hackage-security-lock
Acquired cache lock on
/Users/ur20980/.cabal/packages/hackage.haskell.org/hackage-security-lock
Released cache lock on
/Users/ur20980/.cabal/packages/hackage.haskell.org/hackage-security-lock
Updating index cache file
/Users/ur20980/.cabal/packages/hackage.haskell.org/01-index.cache ...
Index cache updated to index-state 2020-06-28T22:13:53Z
To revert to previous state run:
cabal v2-update 'hackage.haskell.org,2020-06-22T14:34:56Z'
$ ll ~/.cabal/packages/hackage.haskell.org/
total 1561048
drwxr-xr-x 439 ur20980 179427977 14048 Jun 24 17:41 ./
drwxr-xr-x 5 ur20980 179427977 160 Mar 24 18:59 ../
-rw-r--r-- 1 ur20980 179427977 5922515 Jun 22 10:48 01-index.cache
-rw-r--r-- 1 ur20980 179427977 675143168 Jun 22 10:48 01-index.tar
-rw-r--r-- 1 ur20980 179427977 90320136 Jun 22 10:48 01-index.tar.gz
-rw-r--r-- 1 ur20980 179427977 4281050 Jun 22 10:48 01-index.tar.idx
-rw-r--r-- 1 ur20980 179427977 0 Apr 1 09:48 01-index.tar43344-7.gz
-rw-r--r-- 1 ur20980 179427977 4 Jun 22 10:48 01-index.timestamp
drwxr-xr-x 3 ur20980 179427977 96 Apr 5 15:41 Agda/
drwxr-xr-x 5 ur20980 179427977 160 Apr 6 00:16 Cabal/
drwxr-xr-x 3 ur20980 179427977 96 Mar 10 07:59 Diff/
. . . . .
Unfortunately, there doesn't seem to be an option for debugging output to glean more details into why skete
decides theres' nothing to serve.
I'd appreciate guidance how to help you help me to debug and fix this problem.
Thanks!