logo
IRC Archive / Freenode / #opensolaris / 2010 / April / 28 / 10
stallion_work
mbreitba: Infiniband is "stupid fast", IPoIB is like wrapping a PDU in bacon and shoving over IB
TommyTheKid
NFS should use RDMA
lewellyn
i'm afraid to open the attachments :)
TommyTheKid
I am not sure about iSCSI
lewellyn
mmmbacon
balrog
lewellyn: what are the attachments?
mbreitba
they can both use RDMA, and will bypass the IP stack if I can get that configured properly, but I want to at least try to get the IPoIB working
TommyTheKid
i want bacon
mbreitba
plus, the RDMA is even less well documented than the IPoIB
stallion_work
mbreitba: be a man and use RDMA
lewellyn
balrog: a .doc and a .pdf
i've apparently been invited to beijing in july
balrog
hmm.
I see
eviljames
bonsaikitten: I have found my next purchase. Hats off to you! :D
balrog
why are you afraid then? :)
mbreitba
not that IB is well documented at all
wrapster
is there a way i can figure out what internationalization support is provided by default in osol and what can be downloaded later?
rkeene
You won't see the advantages of infiniband over 10GigE if you're using only IP over it.
bonsaikitten
eviljames: really nice hardware, I like them :)
stallion_work
mbreitba: its documented well enough.
lewellyn
because i don't have any ties to the sector, and it was sent with a gmail.com return address over 163's mail servers
mbreitba
the other reason to use IB is that our bladecenter supports it, and the total cost was less than what 10GE would have costed to impliment
balrog
ahh.
open it in google docs :)
TommyTheKid
I would focus on RDMA, although we had talked about using IBoIP because its fanstastically cheaper than 10GIG (I think)
mbreitba
well, I want to get the simpler protocols working first, then move on to the RDMA
lewellyn
i'd still have to download the attachment ;)
huh. bing says it's likely legit
mbreitba
that, and I don't know that the windows boxes that we're currently using work that well with NFSoRDMA
balrog
I don't see how downloading the attachment could cause issues, unless you open it
mbreitba
here's the situation - we've got a large infrastructure running Windows/Hyper-V
Everything is on iSCSI right now, formatted NTFS
we want to move to either Xen or VMWare (haven't decided which yet)
TommyTheKid
mbreitba: I would check the simple stuff first, dladm show-phys, make sure you have link, ifconfig -a .. make sure its UP, make sure it has a proper IP/netmask/broadcast, make sure there is no firewall (ipfilter?)
mbreitba
want to utilize storage on NFS volumes for data portability
TommyTheKid
unfortunately not sure how any of that maps into nexenta :)
mbreitba
Everything that I've checked looks fine
all of the IB queries show that the interfaces are all connected and up
lewellyn
phone
mbreitba
and the nexenta box is able to ping itself on the IB interface
TommyTheKid
make sure the d0ze side has no firewalls as well
mbreitba
no firewalls on the Win side
TommyTheKid
ofcourse if they can ping eachother...
mbreitba
I'm about 20k deep into this project, and it's really starting to piss me off
stallion_work
mbreitba: well... I wouldnt have high hopes for using just NFS
TommyTheKid
so is your interface ibd0?
mbreitba
ibd0 and ibd1 on the nexenta box
stallion_work
you are better off with a NAS head which supports CIFS/NFS access to the same filesystems
mbreitba
well, the nexenta box would be the NAS head end - I can provide CIFS storage out of it also
also - more info about the build here : http://www.zfsbuild.com/ - details about the hardware and intentions
TommyTheKid
ok yuck, our IB router is running S10 :(
IB router -> system with 10G and IB connecitivy, routing between them
mbreitba
I've already tried using bare OpenSolaris - I actually got the IBoIP working ok on it using 2009.06, but it would frequently just drop the connection
iSCSI over the IPoIB link would reset every 35 seconds (literally, like clockwork)
NFS wouldn't mount, but I'm wondering if that was because it was only offering it over RDMA
IPoIB wouldn't work at all in 134a
TommyTheKid
yea, so I wish I had some great knowledge to impart, I know we are messing with IB in our lab, for the same reason you are, but I don't know where we are with it.. I was just asked for a subnet to route to :)
a subnet and a route to it on serverX that is
and the reason I grumbled about S10 was nmap isnt there ;)
mbreitba
well, maybe it's time to jump back into OpenSolaris and hack on it from that end.....the Nexenta stuff is a little more black-box'ish and that seems to be complicating things
lewellyn
TommyTheKid: er... yes it is
install the companion cd :P
TommyTheKid
bah
rkeene
Atleast you aren't trying to do IP-over-Fibre-Channel in Solaris 10. It's completely broken and Sun/Oracle refuses to fix it.
TommyTheKid
I am not going to go installing software on their router
hehe
IBoFC, yikes
lewellyn
your statement is still false ;)
rkeene
IBoFC ?
TommyTheKid
typo
rkeene
Oh
fcip(7D) is completely broken
mbreitba
Nexenta actually told me that the IB card that we had wasn't on the OpenSolaris HCL......which was totally wrong, in any event though, I'm hoping to get something running
TommyTheKid
lewellyn: its not there by default, companion CD (/opt/sfw I assume?) is generally hopelessly out of date, so we stopped installing it
mbreitba
how is the CIFS support on OpenSolaris - is it going to work for my Windows hosts until I can get migrated to Xen/VMWare and use NFS?
TommyTheKid
smrt: explain cifs
smrt
File sharing protocol commonly used on Windows. OpenSolaris includes two cifs servers, one builtin to the kernel, and the samba userspace daemon. (See also: cifs server, samba)
lewellyn
TommyTheKid: it may be out of date, but it is part of a "complete" install. hell, most of the os is hopelessly out of date ;)
TommyTheKid
thats true lewellyn :)
I used to install it once, centrally and mount it NFS.. but I don't know how long its been since I had that :)
mbreitba
and I'm guessing that if I use CIFS, I can export that via NFS later and Xen/VMWare will still be able to read it?
TommyTheKid
sharesmb=yes?
smrt: explain cifs server
smrt
OpenSolaris has a new CIFS (Windows file sharing in SMB) server in the kernel. Good resources for it include http://dlc.sun.com/osol/docs/content/SSMBAG/troubleshooting.html and http://wiki.genunix.org/wiki/index.php/CIFS_Service_Troubleshooting . The CIFS provides better performance and easier administration for CIFS than Samba but Samba provides other SMB services (See also: samba)
lewellyn
mbreitba: if you use the kernel cifs, you can manage nfs and cifs with the same interface
mbreitba
alrighty, I think my answer is to turn back to OpenSolaris then and hack some more on it
also, just out of curiosity, as I didn't find this in my digging - if the NFS server detects RDMA capable hardware, does it only offer NFS over RDMA, or does it offer it up over IP at the same time?
TommyTheKid
I *think* if both ends are on IB, it skips over the IP stack (uses RDMA)
but maybe thats because it doesn't offer NFS over IP on IB interfaces,,, I couldn't tell you there ;)
mbreitba
alrighty - thanks for the info - back to hacking on OpenSolaris - maybe I just need a bigger axe ;)
TommyTheKid
mbreitba: good luck :)
i dont honestly know why they used S10, probably to keep from scaring the OPS folks :p
... apparently either the routing is messed up or they are firewalling, either way, I am not mucking with their config :)
lewellyn
hrm. has anyone packaged tortoisehg for opensolaris yet? it seems overly logical to have ;)
TommyTheKid
lewellyn: quick question... if I build a package with SFE, can I like scp it to another system and install it?
not that I hafe SFE working yet ;)
alanc
lewellyn: GUI's are illogical though
LadySerena
(Action) is a happy kitty
Tonnerre
LadySerena, good!
TommyTheKid
i suppose thats better than beeing a sad salamander ?
LadySerena
At school, I got my pin and senior scrubs today. :D
lewellyn
TommyTheKid: if you build an svr4 package.
alanc: they're still faster than opengrok at getting history ;)
TommyTheKid
lewellyn: otherwise, I can just point at the pkg server that will somehow supposedly be running there on that machine and install it over the network?
lewellyn
and if you're poking around in nautilus, it's fater than opening a terminal and navigating
TommyTheKid: in theory. i've not built ips yet ;)
TommyTheKid
this could be interseting :)
interesting even
lewellyn
remember last night how i was asking about pkgtoolrc? ;)
TommyTheKid
hmm, I was bashing my head on the bootstrap script, so I didnt see it :)
lewellyn
i still don't know why you were editing it.
just run the damn thing :P
TommyTheKid
it doesnt work
lewellyn
smrt: explain doesn't work
smrt
Then make it file for unemployment benefits.
lewellyn
heh
TommyTheKid
hah
I have an NFS home directory and it doesn't cooperate
so I tried to edit the USERHOMEDIR= setting to point at /ws/tm.... and that sorta works, except that it still somehow makes some stuff appear in ~/
lewellyn
yeah. i keep meaning to patch it to deal with nfs homedirs.
but tomww is quiet and i don't know how out-of-date the current download is vs his local copy
TommyTheKid
:)
if he doesn't commit his copy up, then its not there :P
lewellyn
patches are less useful when they're against virtually every line and you're trying to commit them against something fairly different ;)
TommyTheKid
agreed