Official Gamecube IDE/USB interface thread (PROJECT BEGUN!)

Includes PS2, Xbox 1, GameCube (but not the Phantom Game Console)

Moderator:Moderators

Locked
Nonsense Man
Posts:896
Joined:Wed Dec 28, 2005 10:03 am
Location:somewhere
Contact:

Post by Nonsense Man » Fri Sep 07, 2007 7:05 pm

So is this thread dead again or has it come back to life?

r3ap3r
Posts:107
Joined:Wed Jan 24, 2007 8:46 pm

Post by r3ap3r » Sat Sep 08, 2007 2:34 am

Its not dead it has just slowed down because of outside issues but it still is in the works. Cheers^_^

Electric Rain
Senior Member
Posts:1911
Joined:Tue Mar 29, 2005 12:39 pm
PSN Username:Denki_no_Ame
Location:What's it to you? Stalker...
Contact:

Post by Electric Rain » Sat Sep 08, 2007 7:29 pm

It's dead-ish because nobody here ever has the five things needed to complete the project at any given moment: Money, time, knowledge, skill and ambition. :(
Image

r3ap3r
Posts:107
Joined:Wed Jan 24, 2007 8:46 pm

Post by r3ap3r » Sun Sep 09, 2007 2:33 am

I have 4 of the 5 things the one i dont have at the moment is time but real soon it should speed back up to progress.

Electric Rain
Senior Member
Posts:1911
Joined:Tue Mar 29, 2005 12:39 pm
PSN Username:Denki_no_Ame
Location:What's it to you? Stalker...
Contact:

Post by Electric Rain » Sun Sep 09, 2007 3:52 am

Really? You have knowledge and skill? :P

Seriously, do you know what you're doing? What's your next step, anyway?
Image

toru173
Posts:19
Joined:Thu Jun 09, 2005 8:10 am

Post by toru173 » Sun Sep 09, 2007 6:08 pm

wooo, 200th post!

This is incredibly interesting. It's something I've always wanted to do!

The other thing I've considered doing (which is not really on topic) but is to build an SD > IDE adapter, and just plug it into the standard memory card ports. Perhaps this would be a slightly easier alternative? I can imagine it being done with 20 dollars worth of microcontrollers rather then using an expensive FPGA.

All things said and done, though, this is a fantastic idea. A friend of mine was going to give me an FPGA development board so if I can get that off him I'll have a read through the VHDL code (I believe that's what it was) to figure out how this was done, and try to replicate it. My understanding is, though, that the FPGA reads a word through the DD interface, looks up the comparative command (presumably in a lookup table or through just some combinatory logic - I'll have to examine the code) and then outputs the command to the IDE or USB interface. It basically acts as a mediator. Presumably we could modify it such that it's even "smarter" and instead of mediating commands it actually just shuffles blocks of data backwards and forwards. This would need some sort of file system awareness and probably a lot more intelligence, but it has the advantage of being able to move a hard disk directly between the GC and the PC and both would "recognise" the file system. There are a few implementations of FAT and FAT32 that are in the public domain, so this would be wonderful!

User avatar
marshallh
Moderator
Posts:2986
Joined:Sat Sep 10, 2005 2:17 pm
360 GamerTag:marshallh
Location:here and there
Contact:

Post by marshallh » Sun Sep 09, 2007 7:58 pm

There is a good reason destop used a FPGA instead of a simple MCU. The reason is quite simply because the MCU introduces a large propagation delay. You'd have to have a very fast MCU sitting there polling the ports every clock cycle, and even then it will take quite a few clock cycles for a RISC-based cpu to communicate with the other device (presumably IDE.)

SD cards are a poor choice for this sort of thing - first, they use a serial interface that would requires many clock cycles to bit-bang the data over a couple wires, and also data can be read back only in 512byte chunks, which exceeds the RAM capacity of most PICs.

The best way to do this right now is a FPGA and CF card.

Traversing the FAT is quite simple once you get a page in front of you listing offsets for sector data. Basically, you have the FAT (two copies) which stores the starting sector locations of each file and hold the directory structure. Each file is part of a linked-list, that is about 16KB in size, holding the location of the next 16kb piece of the file. That's why it takes so long to calculate free space, especially on older computers.
Image

Electric Rain
Senior Member
Posts:1911
Joined:Tue Mar 29, 2005 12:39 pm
PSN Username:Denki_no_Ame
Location:What's it to you? Stalker...
Contact:

Post by Electric Rain » Sun Sep 09, 2007 9:13 pm

So, here's some questions that I just can't seem to figure out the answers to...

Everywhere I look, including in Destop's Documentation, I see that the 'cube can only read 1.4GB sections. What does this mean for HDDs? Can we put a bootloader on it that lets us select from multiple DOLs that, in total, exceed 1.4GB? Are we supposed to partition the drive in 1.4GB sections and put one DOL to a partition? (The bootloader would obviously take care of pointing the 'cube to the correct partition.) A bootloader is obviously required to boot multiple DOLs (to select which one to boot), and we know it's impossible to boot DOLs bigger than 1.4GB... this is all I'm sure of.

What about if we just wanted one DOL, which actually only took up 1GB or less? Can we use 1GB or smaller CF cards, or does the 'cube need to see all 1.4GB even if it's not all being used? And say there's only one DOL on the HDD/CF card. How does the 'cube know to boot it, actually?

I don't understand much about how the 'cubes filesystem (That's the word, duh!) works. I'm hoping someone can enlighten me...? Unfortunately, I don't think many other people would understand it either, since its filesystem has never been this "open". People have only ever used mini disks or, with a case mod, full sized DVDs with only the inner 1.4GB burnt. We've never really had an xxxGB HDD at our disposal. :?

EDIT: Also, we should have a new person joining our team soon. His name here is "scheiss_freaks". I met him through YouTube from a video of his. I commented, thanking him for showing me "around" GC Linux a bit because I needed to gather as much information as possible for my laptop I'm building. From that, he ended up finding this thread and said he's extremely interested. :) I think he might be able to help us start things back up again.

He directed me to this wiki article, which I somehow never saw. I think it contains all of the same information that the website did, but it does have updated files (straight from nintendo-scene.com... go figure). I think there's a small amount of extra documentation in this package, as well as updated schematics and what may be audio streaming support! :D I have to level with you all, though, I have no idea what that is. :lol: Destop has mentioned that it was one of the only other things he wanted to implement, and that without it, games that use audio streaming won't have any sound. I don't know what these games are... but... *shrug* I'm pretty sure it has something to do with the USB section though. Destop said that the PC audio and the 'cubes audio would have to be mixed together when using audio streaming.

Also, the new schematics/files use a different PROM. This one's only $3.15 at Digikey vs. the old one that's $20.00! Yay for cheaper! 8)
Image

toru173
Posts:19
Joined:Thu Jun 09, 2005 8:10 am

Post by toru173 » Mon Sep 10, 2007 2:53 am

marshallh: I agree. I was only thinking along the lines of an MCU because I've been reading about the Propeller Chip from Parallax. It's basically an 8 core, 32 bit RISC chip clockable up to 120MHz (30 effective, 4 stage pipeline) that can detect a rising edge on every clock (120MHz). It's got 32 IO pins and seems ideal for this sort of situation.

When I spoke of the SD interface I meant using the SPI (EXI?) bus that the memory cards use and building something that allows a standard IDE hard disk to act as a jumbo SD card when looked at by the SPI master (the GC). I've since read over a whole heap more on this subject and the idea was already suggested by ElectricRain - and rejected for beign too slow. You can't boot directly from it, anyway, but it would remove the 1.4 gig limit and also allow for writing to the disk which is something I'm not sure the DD supports.

I mentioned implementing FAT on the disk because in his notes Destop seems to indicate that there is a simple 4meg "image pointer" section, then the rest of the disk is purely DOL images. As such, I got the impression that he didn't actually implement any file system on the disk whatsoever and simply addressed it as a series of blocks. While this is far simpler, it limits the hard disk to only being able to be read on the GC - you can't plug it into your computer and just copy some DOLs over and then expect it to boot. This section of the design, in my opion, needs some refining.

ElectricRain: I saw that guy's youtube video and it's fantastic! When he gets here I'm going to give him a good old pat on the back ^_^

Electric Rain
Senior Member
Posts:1911
Joined:Tue Mar 29, 2005 12:39 pm
PSN Username:Denki_no_Ame
Location:What's it to you? Stalker...
Contact:

Post by Electric Rain » Mon Sep 10, 2007 4:45 am

toru173 wrote:I mentioned implementing FAT on the disk because in his notes Destop seems to indicate that there is a simple 4meg "image pointer" section, then the rest of the disk is purely DOL images. As such, I got the impression that he didn't actually implement any file system on the disk whatsoever and simply addressed it as a series of blocks. While this is far simpler, it limits the hard disk to only being able to be read on the GC - you can't plug it into your computer and just copy some DOLs over and then expect it to boot. This section of the design, in my opion, needs some refining.
Oh, that's a very good point. If it doesn't even have a filesystem, how is it supposed to be read and written to via PC, I wonder? :?

I guess the FPGA must be what tells the GC to load from the "image pointer section" of the drive, right? And I assume this is where the bootloader would go; whether it just points to a single DOL or brings up a menu to select between multiple DOLs.

Speaking of multiple DOLs, is this even possible? If it is, it might just negate the purpose of CF cards. It's a cool concept having one game on a CF card, treating them like DS game cards, but the price per GB is much more expensive for CF cards than it is a 2.5" laptop HDD, and if you can actually put more than one DOL on the drive, it makes a lot more sense to stuff everything on the one drive... and laptop drives aren't even that big. :?

I guess it doesn't matter much yet since the interface for CF cards and IDE HDDs is exactly the same. *shrug* Just something to think about.
Image

toru173
Posts:19
Joined:Thu Jun 09, 2005 8:10 am

Post by toru173 » Tue Sep 11, 2007 2:10 am

It should be pretty simple - just have a "loader" DOL that writes your choice to the bootloader section, and then resets the drive. The bootloader section looks to see what image it should load and starts acting as if that DOL is in the disk drive.

This is all getting a little ahead of ourselves though. What's the use of planning this part of the adapter if we can't get it emulating a disk drive? ^_^

Electric Rain
Senior Member
Posts:1911
Joined:Tue Mar 29, 2005 12:39 pm
PSN Username:Denki_no_Ame
Location:What's it to you? Stalker...
Contact:

Post by Electric Rain » Sun Sep 16, 2007 7:44 pm

I've figured out why this seems like such a daunting task to me. I'm trying to take it all in at once, when I should really be taking it in one step at a time.

First step: r3ap3r, how are the boards coming? The sooner we get some PCBs, the sooner we can start building the actual hardware end of things. PCBs are the first step. Thanks again for offering to send me and TOOHF a free board. (That offer's still on the table, right? :) ) I may redesign the PCB once everything is working, but I have to remember that the prototype comes first. So any PCB will do... it doesn't even matter if it's 30 square inches.

Second step: I need money. :lol: Actually, I may be getting in up to $200 soon, and while I don't imagine I'll need to spend it all on this, I will if I have to. First off, I need new tools. My enormous, corroded, Radioshack soldering iron tip will NOT work for soldering ANY of these parts. It'll find a way to kill the FPGA before I even get 3 pins soldered. :roll: I'll also need a JTAG programmer. I'll probably build this myself to save money.

Third step: After I have the PCB, new tools and a programmer, all that should be left is ordering $100 worth of parts from Digikey, and building this d@mn thing! :twisted:

That's all I'm worried about right now. I need to actually build it and get it over with. I can worry about programming it and not knowing how to load DOLs or the bootloader or WHATEVER ELSE on to the CF card/HDD later. Don't you guys agree?

So, how's everyone else coming along, then? ^-^

Edit: Umm... I think I know everything I need to know. Loading DOLs onto the HDD is still a little foreign to me but I'm sure I can figure it out. However, after printing out the schematic, looking through the files and reading through the Dextrose thread again, I now understand everything about how to stream the ISOs via USB. :shock: I understand so much more of the Dextrose thread now! It's amazing! :D Also, a JTAG programmer is NOT needed. Destop apparently wrote a program that automatically programs the FX2 and the PROM via the FX2's USB port. :twisted: I am much more than 75% sure I can do this now; I'm 95% sure, and I don't see that changing in the near future. 8)

Anyone want to fund me now? I'm sure I can pull it off this time. :)
Image

Electric Rain
Senior Member
Posts:1911
Joined:Tue Mar 29, 2005 12:39 pm
PSN Username:Denki_no_Ame
Location:What's it to you? Stalker...
Contact:

Post by Electric Rain » Mon Sep 17, 2007 5:25 pm

OMFG WIN!!!

I just figured out how to load games onto the HDD. :twisted:

After looking through the files and reading through the Dextrose thread again, I've discovered that Destop apparently made a program to upload to the HDD via the FX2's USB port! :shock:

Coincidentally, I recently decided to start learning C++, so I downloaded Dev-C++ a couple days ago. I don't really know anything about C++ yet, so I didn't even know that the files Destop has in that zip folder are C++ source files. 8)

So, I was now able to open the source files, which before, to me, were just oddly named files with unrecognizable file extensions. I skimmed through some of them, and between the comments inside the files and common sense, I was able to understand a lot more about the software end of this project and what files do what.

Eventually I came across the USB_Writer.c file inside the gc_hdd_storage_tool folder. At the top of this file, there's a comment that says "Hard disk game file storage program"... we're getting somewhere now. :twisted: I remember seeing something during my re-read of the Dextrose thread that said games are downloaded to the drive via the FX2's USB... now I finally understand what that means and how it works.

Knowing NOTHING about compiling any software whatsoever, I tried to compile USB_Writer.c and failed. I tried compiling any file in the gc_hdd_storage_tool folder that I could open in Dev-C++, and they all failed. So, I figured I'd look into the files I couldn't open; the GC_HDD.dsp and the GC_HDD.dsw files. After Googling the file extensions, I found out that this is a "Developer Studio Project" file and "C++ Desktop Settings" file, respectively. They're project files readable by Microsoft Visual Studio 2008; another C++ development program. So, I downloaded and installed that, which took overnight, and woke up early (12:00PM :P ) to try it out.

I tried opening GC_HDD.dsp and it opened right up. After a minute of stumbling around in Visual Studio, I figured out how to build the program (I was looking for the word "compile"... but whatever...). Of course, it failed, complaining that it "cannot open include file 'afxres.h'". So, I went back to Google, trying to find out WTF it was talking about. Apparently I needed to install the "Microsoft Foundation Class Library" (MFC) to get the file. "Where do I find that?", I asked myself. Well, apparently in the "Microsoft Platform SDK". So, I downloaded and installed that with what I thought was the minimum necessary components, and sure enough, the afxres.h file was sitting in "Program Files\Microsoft Platform SDK\Include\mfc", exactly where Google said it'd be. 8) Not wanting to mess around with the project include file settings 'n such in the GC_HDD.dsp project file, I tried to simply copy and paste the afxres.h file into the gc_hdd_storage_tool folder with the other source files, compiled again, and I got a DIFFERENT ERROR MESSAGE!!! (This is actually a good thing! :D ) This one said the same thing, but about a file named Winres.h. Luckily, I was familiar with this file from the research that I did on afxres.h. It was also in "Program Files\Microsoft Platform SDK\Include\mfc", along with afxres.h So, I copied that into gc_hdd_storage_tool as well, re-compiled and...



SUCCESS!!!

It generated the .exe you're looking at below...
Image

Why didn't Destop provide us with an .exe? Why did he make us have to compile it ourselfs? I have no idea, and I don't care, because I JUST COMPILED THE D@MN THING ON MY OWN WITHOUT ANYONE'S HELP!!! MWAHAHAHAHA!!! :twisted:

I've uploaded the .exe to my server for you guys to download and mess with, since most of you are probably too lazy to go through all the steps necessary to compile it yourself. :wink:

WARNING: The Add GAME and Add LOADER buttons don't do anything until you open a drive. If you open an important hard drive just to see how the Add GAME and Add LOADER buttons work, DO NOT actually load something. Make sure you hit cancel at the file open prompt.

I made the mistake of loading a loader, and it wreaked havoc on my hard drive's MBR, instantly freezing my computer and preventing it from booting up. This is the second time my MBR's gotten f'ed, so I sorta knew what to do. It wasn't quite the same as last time and it was less severe, but it still took me a good hour to fix it. (Norton Disk Doctor from "Hiren's Boot CD" is a godsend.) Think about it... consider this bit of info:
Destop in the Dextrose thread wrote:I might need to explain some more about how I laid out the file system on the hardrive.
First 4 megs are the iso image of the loader.
next 4 megs is the file table:
typedef struct {
char name[32];
UINT LBA1; //high
UINT LBA2; //low
UINT res1;
UINT res2;
} FILE_ENTRY;


I was gonna use the first 2 megs of the file table for the entries and the next 2 for stuff like action replay codes hence the reserved pointers.

I just let each game take up 0x57158000 so you can tac on some trainer/ intro to the game if the iso is full. Will the gamecube drive let you access more the standard 1.4gig size? I dont think it will.
So basically, he built his own filesystem. It's not FAT32, it's not NTFS... it's totally custom. All new hard drives come unformatted, with no filesystem in place. They are known as RAW. Presumably, this is how it would be installed in the IDE to DD adapter. This newly discovered program is the only thing that can actually write stuff to the hard drive, so it clearly must write the custom filesystem to it as well. So, if you apply the program to a formatted hard drive with an existing filesystem, well... I guess I found out what would happen then. :lol:

The worst part is that I already had this entire message typed up when I loaded the loader and my computer froze up. :evil: But, that's okay... I've lost long posts and emails before, and they always end up worded better the second time I write them anyway. *shrug*

So, what do you guys think? Pretty exciting, no? It looks like EVERYTHING is figured out... in theory. We don't know how everything's gonna pan out once we actually get the hardware built and start trying to program chips and install files, but at least we know what we're supposed to do. :) Is anyone still in the dark about anything? I think I can answer any questions you may have; feel free to ask them.

P.S. I uploaded the most recent version of "ideusb.zip" (now "GC_IDE_ATA.zip") linked to in the wiki article to my server so you guys don't have to sign up with the nintendo-scene.com forums to download it. :wink:
Last edited by Electric Rain on Mon Sep 17, 2007 5:51 pm, edited 2 times in total.
Image

User avatar
marshallh
Moderator
Posts:2986
Joined:Sat Sep 10, 2005 2:17 pm
360 GamerTag:marshallh
Location:here and there
Contact:

Post by marshallh » Mon Sep 17, 2007 5:37 pm

Heh, compiling other people's projects is always fun.

I opened up GC_HDD.dsw in my copy of VC++6 and it compiled right up. Must be because your copy is too new and doesn't include MFC :P

Now your next step is understanding the code. I have barely looked at it myself but I'd guess it will load the FX2 with firmware upon bootup. If you need help with any of the FX2-related usb stuff, I can help...
Image

toru173
Posts:19
Joined:Thu Jun 09, 2005 8:10 am

Post by toru173 » Tue Sep 18, 2007 7:13 am

This is where I do my "I told you so" dance - it's not even really a filesystem, it's just a list of "FILE_ENTRY" structs which are pointers to the files. You quoted from the post I was refering to earlier, by the way

Anyways, I might might might might be getting an FPGA dev board soon. Might. IF I am, I could have a go at "porting" his vhdl to my dev board - this would allow me to experiment without having to get a pcb done. Dev boards are expensive though so I don't recommend this route, it just so happens a friend of a friend doesn't want his anymore and he's stonking rich so he doesn't care about the cost.

Locked