Not signed in (Sign In)

Welcome, Guest

Want to take part in these discussions? Sign in if you have an account, or apply for one below

Vanilla 1.1.5a is a product of Lussumo. More Information: Documentation, Community Support.

    •  
      CommentAuthorniten
    • CommentTimeJul 7th 2006 edited
     

    1 The Case for the Migration

    • celery has been hacked, thrice.
      • At least once, the hacker got full root access, though I caught him quickly.
      • We can’t really trust celery anymore.
      • celery is having weird software problems
      • /var occasionally remounts readonly, due to filesystem corruption, effectively ‘crashing’ the system
      • grub doesn’t always work, which is ridiculous
      • celery often loses it’s internet connection for an hour or so, for no good reason
        • thecitadel and starkhorn stay connected
      • It doesn’t reboot
    • During the move, we could cleanly implement stuff that’s been on the backburner for way too long:
      • LDAP users (that is, no non-LDAP users), Kerberos authentication
      • MyDNS
      • Proper filesystem security, possibly SELinux, ACLs
      • Proper host-based authentication
    • What else is pending?

      2 The Hosts

      We have the following hosts to play with:

      • celery
    • quasar
    • starkhorn
    • pinky

      Here’s what I have in mind:

      host role
      celery User server
      quasar Webpages, other ‘critical’ stuff
      starkhorn Master LDAP & Kerberos server, admin server
      pinky Slave LDAP & Kerberos server
    • celery:
      • User’s home directories, email, webpages, are stored here
      • Webmail
      • DNS Slave server
      • A development server
      • Runs a Jabber node
    • Many users with shell access
    • quasar
      • Standalone pages (retpa, ovfp, fudo.org, forums)
      • DNS Master server
      • Also runs a Jabber node
    • Few users with shell access (administrators & developers, no users)
    • starkhorn
      • LDAP Master server
      • Kerberos Master server
    • Minimal shell access, and not through LDAP (still uses Kerberos, though), using passwords and ssh keys
    • pinky
      • LDAP Slave server
      • Kerberos Slave server
    • Essentially no shell access (only root, and no remote access, only for troubleshooting the daemons)

      3 Topics for Consideration

      1. Security
        1. Filesystem security—improved Unix-style (user, group), ACLs, SELinux
        2. Host security—SSH, PAM, LDAP?
    1. Resources—limits.conf, quotas
    • CommentAuthorSwaff
    • CommentTimeJul 8th 2006
     
    Some thoughts:

    1. Don't forget about upbringer.
    2. Pinky doesn't have a net connection (Ark has an extra IP, doesn't he?)
    3. If ovfp and retpa are going to be on quasar, they'll need, at least, FTP access, if not shell access.
    4. Any harm in having all machines host a DNS and IRC server?
    •  
      CommentAuthorniten
    • CommentTimeJul 8th 2006
     
    1. Isn’t upbringer a desktop computer? Like, doesn’t your mom use it? I guess it doesn’t really matter, but I left thecitadel & darkstar off the list, just because that way we can take them down as necessary, and whatever. But for IRC or DNS, definitely.
    2. It’s pointless (well, almost) to put pinky online, unless it’s right beside quasar/darkstar. It’s a backup authentication server…so if starkhorn AND pinky go down, then nobody could log into quasar. Kinda weird. It also seems as though Ark also gets disconnected whenever I do. At the very least, it should be somewhere else in the city—at my sister’s place, or something.
    3. Yeah, sure, that’s fine. They’ll only be doing periodic, simple updates, so giving them limited access will be no problem, and they’re unlikely to break the server.
    1. celery and quasar could certainly be DNS & IRC. starkhorn & pinky are strictly LDAP & Kerberos—anything else is a potential vulnerability (especially with trashy IRC servers…)
    • CommentAuthorSwaff
    • CommentTimeJul 9th 2006
     
    My parents have an extra IP as well, I would think. We'd just need a hub I guess.
    •  
      CommentAuthorniten
    • CommentTimeJul 11th 2006 edited
     

    Okay, first step is complete! Check this out: all users are now LDAP-ized.

    Well, all Fudo users are in a database on starkhorn. First step in the migration.

    Now, let’s get quasar using this database. Then, start moving files.

    Hey, we could get our main pages serving off of quasar then bring celery to quasar, instead of vice-versa.

    • CommentAuthorreaper
    • CommentTimeJul 18th 2006 edited
     
    I'm ready to go. The amout of user data will be a bit of a problem.

    Also, i see little to no mention of XMPP or email, both of which are major services we provide.

    The first order of business is to unload celery, and rebuild it. Rebuild the software on it, i should say, right from the OS, up.

    I'll see if i can summon niten tonight.
    •  
      CommentAuthorniten
    • CommentTimeJul 19th 2006
     

    XMPP is straightforward. I’m pretty sure, anyway. Basically, just move it over, then update DNS.

    Email…well, configuring exim will be…fun…but it may be as simple as just moving the config right on over.

    Hrm. only issue here is maildir vs mbox. But that’s an issue and a half. We’ll have to make sure /all/ mail progs/daemons are using maildir.

    For users, we have to:

    1. Get LDAP users working on quasar, which is something I’ve gotten pretty good at
    2. Move home directories
    3. Change filesystem permissions from the old UIDs on celery to the new UIDs on quasar. one nice thing about this is that anybody who was /already/ in LDAP will have the same UID—and that’s most of us regular users.
    1. Test, basically. Especially, make sure email works.
    •  
      CommentAuthorniten
    • CommentTimeJul 27th 2006
     

    Damn it.

    LDAP on celery is busted. The database corrupted itself. Again.

    I think—that is, I think—that celery itself is to blame. God knows, /var has given me no end of trouble. bdb hasn’t impressed me much, though; it can’t seem to get un-corrupted, once it is. Not a good sign. I mean, we use a single bdb database on celery, for LDAP, and it’s failed us—_twice_. And once, I believe, on starkhorn. And every time, the recovery failed. Meanwhile, MySQL and PostGreSQL, both of which store data on /var, have—while occasionally experiencing corruption—silently corrected any problems.

    We could switch to ldbm, though apparently that requires a bit more consideration—%{font-family: fixed-width}bdb% is intended for exactly the kind of simple tasks that LDAP requires, whereas ldbm is a little more…database-y.

    Or we could use MySQL or PostGreSQL, as a backend for LDAP. Or—*or*—we could simply use MySQL or PostGreSQL to store our users.

    Egads. All my work would be in vain! Plus, lots of stuff supports LDAP; it’s kind of the universal standard for storing distributed databases of users/user information. All email clients, many services (ejabberd, for example)....web services like dokuwiki…

    But it can’t keep corrupting itself like this! It’s ridiculous!

    No, I think LDAP is the right choice. But I’m going to make a goddamned cron script to back the database up daily—both the LDIF, and the actual database files. They won’t be overly large. Plus, I’m going to switch to ldbm, I think.

    Anyway, just musing.

    • CommentAuthorreaper
    • CommentTimeAug 4th 2006
     
    I thought you were saying that LDAP was happening on starkhorn, not celery. Why is there a bdb LDAP database on celery?

    And we need a better pinky, i think. I guess pinky can work for things, but i'd want it to run the most minimal set of services it possibly can. A P-120 with 24MB of RAM is pretty limited.

    It could probably handle SSH, ntpd, and an LDAP server, but that's it. I guess i slam pinky pretty hard, but that's because i was trying to get it to run many services (HTTP, DNS, NFS, smokeping, IRC) and it just ran out of memory. Pinky is still slow, but if we only give it 1 main task, it will probably be okay.

    The other thing i will mention here is that scott said that it was okay for me to put a personal server on the rainy day network. Even at the datacentre, he was suggesting. Pinky could be that box. And if it were, i might just put it at the office, to prevent it from being embarrassed at the datacentre.
    •  
      CommentAuthorniten
    • CommentTimeAug 4th 2006 edited
     

    LDAP is happening on starkhorn. Meanwhile, though, it’s on celery because it’s been on celery for ages.

    That’s what I did, though: I pointed pam_ldap and nss_ldap on celery towards starkhorn. It’s working, now…albeit weirdly. There’s lots of dual entries, because every user is in starkhorn, but many are still in celery:/etc/passwd. The system uses passwd first.

    BTW, the last wave of migrations failed. All the users who didn’t manually change their passwords earlier has a broken LDAP profile. That sucks. I was sure it was gonna work.

    So…I should generate passwords. I am going to generate passwords. It’s a sonofabitch, though. Oh, and users should be able to change their generated passwords, and that means creating a simple PHP website…

    Oh, and about Kerberos, here’s how I think it’s going to go down: normal users will have a single password, stored in LDAP and not terribly secure (maybe a little less secure than /etc/passwd…well, or maybe a little bit more, I dunno, depends how careful I am). They will not have shell access, or any other access to low-level stuff. Any service that is both a) secure and b) uhrm, mission critical (?) will use Kerberos. That means you and I have two passwords. Which, at first, annoyed me; but then I thought about it, and that’s really not a bad idea. That way we can check our email and FTP things around on public networks without fear of our passwords being sniffed.

    Alright: now, the adventures of pinky at the great datacenter! Here’s what I say it should run:

    • DNS, because it can, and because it’ll have a fixed IP
    • Kerberos, because we need a backup Kerberos server, and because a Kerberos server shouldn’t be doing anything else, anyway!
    • LDAP, if there’s enough RAM left. By the way, I have old RAM we can try, now that pinky’s down and out anyway. LDAP isn’t vital, though, because any ol’ server can run LDAP.

      Come to think of it, though, we’ve got no excuse to use pinky. I have a P133 (I think) and a P100 sitting in my apartment, unused. Either of those would be a step up, albeit a slight one. We’ll have to check how much RAM they have.

    • CommentAuthorSwaff
    • CommentTimeAug 4th 2006
     
    Pinky's a P-100, and if I recall correctly the RAM that's in it is 2x4MB and 2x8MB, and those are 72 pin SIMMs.

    I think, reaper, that if we had removed apache and smokeping, and maybe NFS, from Pinky before, we could have kept DNS and IRC without any problems.
    •  
      CommentAuthorniten
    • CommentTimeAug 4th 2006
     

    Naw, it was hacked. As I recall, it was running mystery procs for days.

    • CommentAuthorreaper
    • CommentTimeAug 5th 2006
     
    yes, pinky was compromised, that's it's main reason for being pulled down.

    I guess we should take stock of all hardware we have and build the baddest system we can.

    I could probably scam some old RAM from work, though i'm not sure w've got 72-pin SIMMs, those will be a little scarce these days.

    Also, the two passwords (insecure and secure) make some sense. Especially if we want to protect our good passwords, passing them via FTP or other insecure protocols is foolish. So it will be okay, i think.
    •  
      CommentAuthorniten
    • CommentTimeAug 5th 2006
     

    Okay, I was thinking to myself, and I thought I might as well put it here.

    Let’s walk through a couple scenarios, shall we?

    • Login to check email:
      1. Establish connection (i.e. get by the firewall)
      2. Check whether the given user is a member of host or host_email
      3. If he is, check his password against LDAP
    1. If the password matches, then allow access.

      Simple enough.

      • Login by SSH
      • Establish connection
      • Check whether the givne user is a member of host or host_shell
    1. Authenticate against Kerberos

      Wait, this is simpler than I thought. So, instead, a list of services and their auth method:

      • httpd
    • LDAP, except for any admin interface, which must be over SSL
    • imap
    • LDAP
    • login
    • Kerberos
    • FTP
    • LDAP
    • SFTP
    • Kerberos (but this will match ssh, anyway)
    • SSH
    • Kerberos
    • sudo
    • Kerberos

      That’s the only ones that stand out of celery:/etc/pam.d.

      This can be done fairly easily with a common-auth file and a secure-auth file.

      There’s one final case I’m working on: host communication. How do the hosts communicate with each other? For example, I wanna put /fudo on every host via SVN. SVN uses SSH to communicate. So…should root be able to ssh from host to host, sans password? Not a good idea. A separate ‘fudo’ user, with access to all the necessary files? That’s a better idea. With an ssh key…is that good enough? Or should we use Kerberos?

      I’m going to get started and make fudo users on quasar and starkhorn, anyway…

  •  
    Why are we not using SSL for webmail logins?
    •  
      CommentAuthorniten
    • CommentTimeAug 7th 2006
     

    Well, because…hrm. It’s kind of complicated. Wait, no it isn’t: SSL doesn’t work with subdomains. Yet…it’s part of the standard now, but it’ll take a few years before the various browsers and servers implement it. In the meantime, if you use webmail.fudo.org, you get all sorts of errors (well…a few, anyway).

    Maybe during the move we should move everything to www.fudo.org/webmail, www.fudo.org/forums, etc? webmail.selby.ca could just be a referral page to https://www.fudo.org/webmail/

  •  
    Could you elaborate a bit more as to why subdomains don't work?
    •  
      CommentAuthorniten
    • CommentTimeAug 7th 2006
     

    Well, sure, here goes…

    You request something from the server celery.fudo.org. Specifically, you request ‘webmail.fudo.org’. The server gets the request, which is _addressed_to celery.fudo.org, in the packet. It doesn’t know what the request is for, since the request itself is encrypted. That is, the packet is encrypted; so the actual “can you please send me the contents of the page, ‘webmail.fudo.org’” is hidden. It doesn’t know which certificate (aka key) to use to decrypt the packet, since it doesn’t know what site (i.e. webmail.fudo.org) the packet is headed for, until it can decrypt the packet and find out. Catch-22.

    So, the only thing it can do is decrypt the packet using it’s own certificate, for the host celery.fudo.org. However, part of the spec of SSL is that each, er, subdomain/host should have it’s own key. The name ‘celery.fudo.org’ is encoded right there in the SSL certificate. So, when you attempt to connect to webmail.fudo.org using HTTPS you get a message saying, “warning: you attempted to contact webmail.fudo.org, but the remote certificate was issued to celery.fudo.org. Do you want to: 1) accept permanently, 2) accept for this session, 3) reject this certificate.

    Or, it works something very similar to that, anyway. So for now, hosts-to-certificates is a 1-to-1 mapping, but unfortunately (and especially for us), subdomains-to-hosts is very far from 1-to-1. If we had one host for every domain (or, at the very least, one server for every subdomain—i.e. it might be possible, if somewhat insane, to run different subdomains on different ports, and get a similar effect), then each host/server could have it’s own, unique certificate, and everything would just work fantastically. webmail.fudo.org would then be a distinct server with a distinct IP and a different certificate. If that were the case, then the request would go directly to webmail.fudo.org, which would be able to successfully decrypt the packet, and the hostname in the cert would match the actual hostname.

  •  
    I see what you mean now. Only one domain/subdomain would work on port 443. Unlike 80 where you can have multiple domains/subdomains. Correct? AND even if you did make your own certs you would still get a warning because the issuer would not be a recognized "trusted" authority.
    •  
      CommentAuthorniten
    • CommentTimeAug 8th 2006
     

    Re: The “trusted authority” issue: That’s true. You only have to accept once, though. So, the first time you ever went to www.fudo.org it would complain about not trusting the issuer, but you could chose “Always accept certs signed by this issuer (fudo.org)”, and it should never bother you again.

    And, for the second part…that’s basically right. The actual port number is irrelevant, but with SSL-encrypted HTTP, the server can only handle the actual, literal name of the host, not a DNS alias. So, only one—and you couldn’t use www.fudo.org, you’d have to use celery.fudo.org (unless the actual hostname of the machine was ‘www’). No subdomains, unless those subdomains were also actual, literal hosts (so webmail.fudo.org would have to be a different machine altogether).

    Now, if everything was stored at celery.fudo.org (so: https://celery.fudo.org/www/, https://celery.fudo.org/webmail/ and https://celery.fudo.org/forum/), it would work.

    The reason it works like that is because the internet, as originally envisioned, was never supposed to have dozens of aliases pointed to a single host. The subdomain (aka hostname)-to-host ratio was supposed to be 1-to-1.

    Heh, encryption is weird.

    • CommentAuthorJoker4ever
    • CommentTimeAug 8th 2006 edited
     
    I disagree with you there niten. The default port for ssl is 443. Take a look at this https://webmail.getjacked.ca https://getjacked.ca takes you to the same site but http://getjacked.ca takes you to a different site. You can make it listen on different ports and what not. If you were to make it listen on a port number other than the default then you would have to you would have to do https://address:portnumber. http://getjacked.ca and https://webmail.getjacked.ca are the same machine. So either I disagree with your or I just don't understand what you said. I hope this isn't confusing but I have to post fast as I'm at work.
    •  
      CommentAuthorniten
    • CommentTimeAug 8th 2006
     

    Yeah, if you change the port number, you have to specify it. But you could (I think, and only with great difficulty) run several servers, on different ports, with different certs…..

    Now that I think about it, though, no, that wouldn’t work, at least without some weird iptables magic or some shit. It still wouldn’t match the DNS hostname.

    You can have a single site served by SSL, that’s true. In your example, there’s no problem with having http://getjacked.ca and https://webmail.getjacked.ca. And you could, additionally, have any number of subdomains that you wanted. However, you couldn’t have any other subdomain which used SSL. Which is okay if you’ve only really got the one thing you want to protect, but we’d want to protect all the places we use our passwords.

    Or, heh, if you don’t mind getting the occasional error message, it’s fine to do it for all domains. We’ve been amazingly reticent about this, really… If you go to gmail, or Yahoo mail, or encrypted Microsoft pages, you get the exact same error (the first time). Whenever you click “Accept this certificate perminently”, you’re accepting a different hostname. But, well, that’s fine for your Google or your Microsoft, but we insist on doing it right, goddamn it!

  •  
    Yes that's what I was getting at in my original post.
    • CommentAuthorreaper
    • CommentTimeAug 9th 2006
     
    There's one other way they do multiple SSL sites on a single host. Multiple IPs. So each site has it's own DNS name, which points to unique IPs, on which the reverse lookup matches. That's how the big boys do it. They certainly aren't forced to have a different computer for each SSL website.

    It's kinda cool, then, because you can do IP-based virtualhosting, instead of name-based.

    But you guys have gotten heavily sidetracked here. Aren't we talking about migration? And yeah, having a consistent SSL cert for www.fudo.org would be cool. then we could just encrypt all the stuff behind that, like webmail, forums, or w/e. I agree with that idea.
    •  
      CommentAuthorniten
    • CommentTimeAug 11th 2006
     

    For that to work, though, we’d have to keep everything under celery.fudo.org, or make www the cname for celery, yes?

  • 1 to 25 of 25

    1.