overview for moonshine69

Three reasons why "AI" is not conscious and perhaps never will be by moonshine69 in c/technology@lemmy.world

[-] moonshine69 1 points 3 days ago

And the earth couldn't possibly revolve around the sun.

They have completely recovered the wiring of fruit fry brains and trained models on that architecture..is a fruit fly conscious? Is a little device with all th same movement and sense capability using the fruit fly con nectome conscious

Is the Cornucopia used by the Fruit of the Loom brand actually a Mandela Effect or is it like, proven they had that as their logo for a real actual period of time before they ditched it? What sorcery by moonshine69 in c/nostupidquestions@lemmy.ca

[-] moonshine69 2 points 4 days ago

Computer memory (main memory anyway) is rewritten every time it's read ...nay even more so it's refreshed even when it's not read

And flash drives, you probably aren't getting a decade out of one of them without bit flips

With funding, we could be rid of all Cane Toads. by moonshine69 in c/australia@aussie.zone

[-] moonshine69 6 points 1 week ago

Ya looks like it, the density of toads and snakes is much higher in real life

Never buy a bad gram again. (date unknown, suspected 1970s) by moonshine69 in c/vintageads@sh.itjust.works

[-] moonshine69 7 points 3 weeks ago

You missed your meth thorns

How hard is it really to have your own webcrawlers and index? by moonshine69 in c/asklemmy@lemmy.world

[-] moonshine69 5 points 1 month ago

Oops, realized I didn't answer your question about actually crawling, dig into common crawl documentation, they provide a bunch of technical data and stats that show you the scale..2-4billion pages per month

And note CC just does a sample of the pages it finds. So the more monthly dumps don't contain all of the data afaik

And the number above are for one of the monthly dumps

https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlermetrics

How hard is it really to have your own webcrawlers and index? by moonshine69 in c/asklemmy@lemmy.world

[-] moonshine69 12 points 1 month ago

The index (i.e just the list of urls) of common crawl is 250gb. The actual crawl is 500tb ish. Then pull out just the text you get down to 30tb ish. And then you need that all to be in a fts index.. (something like opensearch/elasticsearch) idk how big the cluster you would need for that is.. maybe 2tb per data node max. So 15 pretty beefy vms (8cpu w/64gb memory) with 2tb nvme for data and a few more as controller nodes

Common crawl numbers pulled from this paper

https://aclanthology.org/anthology-files/anthology-files/pdf/L/L18/L18-1286.pdf

Cluster sizing numbers pulled from tea leaf divination